2023-04-17 07:59:39,079 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d 2023-04-17 07:59:39,092 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-17 07:59:39,113 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-17 07:59:39,114 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18, deleteOnExit=true 2023-04-17 07:59:39,114 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-17 07:59:39,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/test.cache.data in system properties and HBase conf 2023-04-17 07:59:39,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/hadoop.tmp.dir in system properties and HBase conf 2023-04-17 07:59:39,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/hadoop.log.dir in system properties and HBase conf 2023-04-17 07:59:39,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-17 07:59:39,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-17 07:59:39,118 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-17 07:59:39,259 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-17 07:59:39,594 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-17 07:59:39,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-17 07:59:39,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-17 07:59:39,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-17 07:59:39,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 07:59:39,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-17 07:59:39,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-17 07:59:39,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 07:59:39,600 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 07:59:39,600 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-17 07:59:39,600 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/nfs.dump.dir in system properties and HBase conf 2023-04-17 07:59:39,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/java.io.tmpdir in system properties and HBase conf 2023-04-17 07:59:39,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 07:59:39,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-17 07:59:39,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-17 07:59:40,104 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 07:59:40,109 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 07:59:40,519 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-17 07:59:40,683 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-17 07:59:40,704 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 07:59:40,755 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 07:59:40,795 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/java.io.tmpdir/Jetty_localhost_localdomain_39001_hdfs____.up8wm3/webapp 2023-04-17 07:59:40,947 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:39001 2023-04-17 07:59:40,982 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 07:59:40,982 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 07:59:41,500 WARN [Listener at localhost.localdomain/46073] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 07:59:41,604 WARN [Listener at localhost.localdomain/46073] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-17 07:59:41,625 WARN [Listener at localhost.localdomain/46073] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 07:59:41,633 INFO [Listener at localhost.localdomain/46073] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 07:59:41,639 INFO [Listener at localhost.localdomain/46073] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/java.io.tmpdir/Jetty_localhost_45797_datanode____.ysqmcl/webapp 2023-04-17 07:59:41,724 INFO [Listener at localhost.localdomain/46073] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45797 2023-04-17 07:59:42,000 WARN [Listener at localhost.localdomain/40407] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 07:59:42,776 DEBUG [Listener at localhost.localdomain/40407] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d 2023-04-17 07:59:42,780 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcd920b6cc1cb4254: Processing first storage report for DS-d65c3e09-dd61-4a6c-844c-cef43ed99016 from datanode 372b2a2d-65cf-4d91-a317-aff28cf3703e 2023-04-17 07:59:42,781 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcd920b6cc1cb4254: from storage DS-d65c3e09-dd61-4a6c-844c-cef43ed99016 node DatanodeRegistration(127.0.0.1:42287, datanodeUuid=372b2a2d-65cf-4d91-a317-aff28cf3703e, infoPort=45903, infoSecurePort=0, ipcPort=40407, storageInfo=lv=-57;cid=testClusterID;nsid=250919715;c=1681718380168), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-17 07:59:42,782 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcd920b6cc1cb4254: Processing first storage report for DS-a3cdd981-cc84-4053-8563-1a9740f4b223 from datanode 372b2a2d-65cf-4d91-a317-aff28cf3703e 2023-04-17 07:59:42,782 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcd920b6cc1cb4254: from storage DS-a3cdd981-cc84-4053-8563-1a9740f4b223 node DatanodeRegistration(127.0.0.1:42287, datanodeUuid=372b2a2d-65cf-4d91-a317-aff28cf3703e, infoPort=45903, infoSecurePort=0, ipcPort=40407, storageInfo=lv=-57;cid=testClusterID;nsid=250919715;c=1681718380168), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-17 07:59:42,826 INFO [Listener at localhost.localdomain/40407] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18/zookeeper_0, clientPort=62477, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-17 07:59:42,840 INFO [Listener at localhost.localdomain/40407] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62477 2023-04-17 07:59:42,849 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:42,852 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:43,470 INFO [Listener at localhost.localdomain/40407] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36 with version=8 2023-04-17 07:59:43,470 INFO [Listener at localhost.localdomain/40407] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/hbase-staging 2023-04-17 07:59:43,774 INFO [Listener at localhost.localdomain/40407] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-17 07:59:44,158 INFO [Listener at localhost.localdomain/40407] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=45 2023-04-17 07:59:44,187 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,187 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,188 INFO [Listener at localhost.localdomain/40407] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 07:59:44,188 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,188 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 07:59:44,315 INFO [Listener at localhost.localdomain/40407] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 07:59:44,378 DEBUG [Listener at localhost.localdomain/40407] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-17 07:59:44,468 INFO [Listener at localhost.localdomain/40407] ipc.NettyRpcServer(120): Bind to /188.40.62.62:34569 2023-04-17 07:59:44,478 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:44,481 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:44,502 INFO [Listener at localhost.localdomain/40407] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34569 connecting to ZooKeeper ensemble=127.0.0.1:62477 2023-04-17 07:59:44,589 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:345690x0, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 07:59:44,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34569-0x10196577f520000 connected 2023-04-17 07:59:44,667 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 07:59:44,668 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 07:59:44,671 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 07:59:44,680 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34569 2023-04-17 07:59:44,680 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34569 2023-04-17 07:59:44,681 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34569 2023-04-17 07:59:44,684 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34569 2023-04-17 07:59:44,685 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34569 2023-04-17 07:59:44,692 INFO [Listener at localhost.localdomain/40407] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36, hbase.cluster.distributed=false 2023-04-17 07:59:44,756 INFO [Listener at localhost.localdomain/40407] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-04-17 07:59:44,757 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,757 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,757 INFO [Listener at localhost.localdomain/40407] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 07:59:44,757 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 07:59:44,757 INFO [Listener at localhost.localdomain/40407] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 07:59:44,762 INFO [Listener at localhost.localdomain/40407] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 07:59:44,765 INFO [Listener at localhost.localdomain/40407] ipc.NettyRpcServer(120): Bind to /188.40.62.62:39843 2023-04-17 07:59:44,768 INFO [Listener at localhost.localdomain/40407] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-17 07:59:44,773 DEBUG [Listener at localhost.localdomain/40407] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-17 07:59:44,774 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:44,778 INFO [Listener at localhost.localdomain/40407] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:44,780 INFO [Listener at localhost.localdomain/40407] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39843 connecting to ZooKeeper ensemble=127.0.0.1:62477 2023-04-17 07:59:44,791 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:398430x0, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 07:59:44,794 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 07:59:44,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39843-0x10196577f520001 connected 2023-04-17 07:59:44,796 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 07:59:44,797 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 07:59:44,800 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39843 2023-04-17 07:59:44,800 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39843 2023-04-17 07:59:44,801 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39843 2023-04-17 07:59:44,808 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39843 2023-04-17 07:59:44,808 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39843 2023-04-17 07:59:44,810 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:44,825 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 07:59:44,827 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:44,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 07:59:44,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 07:59:44,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:44,850 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 07:59:44,851 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,34569,1681718383610 from backup master directory 2023-04-17 07:59:44,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 07:59:44,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:44,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 07:59:44,864 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 07:59:44,864 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:44,866 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-17 07:59:44,868 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-17 07:59:44,952 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/hbase.id with ID: 1753a648-9158-4d24-a499-11a29f8becb8 2023-04-17 07:59:45,008 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 07:59:45,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:45,064 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1839b432 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:45,109 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c7c7b7c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:45,131 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 07:59:45,133 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-17 07:59:45,149 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-17 07:59:45,149 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-17 07:59:45,152 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 07:59:45,155 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 07:59:45,157 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 07:59:45,187 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store-tmp 2023-04-17 07:59:45,215 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:45,215 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 07:59:45,216 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 07:59:45,216 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 07:59:45,216 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 07:59:45,216 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 07:59:45,216 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 07:59:45,216 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 07:59:45,218 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/WALs/jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:45,243 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34569%2C1681718383610, suffix=, logDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/WALs/jenkins-hbase16.apache.org,34569,1681718383610, archiveDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/oldWALs, maxLogs=10 2023-04-17 07:59:45,299 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK] 2023-04-17 07:59:45,308 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 07:59:45,368 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/WALs/jenkins-hbase16.apache.org,34569,1681718383610/jenkins-hbase16.apache.org%2C34569%2C1681718383610.1681718385253 2023-04-17 07:59:45,369 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK]] 2023-04-17 07:59:45,370 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-17 07:59:45,370 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:45,373 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,374 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,445 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,453 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-17 07:59:45,476 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-17 07:59:45,487 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:45,493 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,495 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,510 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 07:59:45,515 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 07:59:45,516 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70617710, jitterRate=0.05228587985038757}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 07:59:45,517 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 07:59:45,519 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-17 07:59:45,538 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-17 07:59:45,538 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-17 07:59:45,540 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-17 07:59:45,542 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-17 07:59:45,571 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 29 msec 2023-04-17 07:59:45,572 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-17 07:59:45,597 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-17 07:59:45,603 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-17 07:59:45,625 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-17 07:59:45,628 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-17 07:59:45,630 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-17 07:59:45,636 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-17 07:59:45,640 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-17 07:59:45,649 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:45,651 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-17 07:59:45,651 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-17 07:59:45,662 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-17 07:59:45,694 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 07:59:45,694 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 07:59:45,694 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:45,695 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase16.apache.org,34569,1681718383610, sessionid=0x10196577f520000, setting cluster-up flag (Was=false) 2023-04-17 07:59:45,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:45,757 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-17 07:59:45,759 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:45,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:45,808 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-17 07:59:45,809 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:45,811 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.hbase-snapshot/.tmp 2023-04-17 07:59:45,908 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-17 07:59:45,912 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(952): ClusterId : 1753a648-9158-4d24-a499-11a29f8becb8 2023-04-17 07:59:45,916 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-17 07:59:45,919 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-17 07:59:45,919 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-04-17 07:59:45,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:45,921 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681718415921 2023-04-17 07:59:45,923 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-17 07:59:45,926 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-17 07:59:45,926 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-17 07:59:45,927 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 07:59:45,927 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-17 07:59:45,932 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 07:59:45,934 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-17 07:59:45,935 DEBUG [RS:0;jenkins-hbase16:39843] zookeeper.ReadOnlyZKClient(139): Connect 0x712c8aa1 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:45,935 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-17 07:59:45,942 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-17 07:59:45,942 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-17 07:59:45,943 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-17 07:59:45,943 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-17 07:59:45,946 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:45,947 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-17 07:59:45,949 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-17 07:59:45,949 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-17 07:59:45,950 DEBUG [RS:0;jenkins-hbase16:39843] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7feb6af4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:45,951 DEBUG [RS:0;jenkins-hbase16:39843] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bd4e1d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-04-17 07:59:45,955 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-17 07:59:45,956 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-17 07:59:45,958 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1681718385958,5,FailOnTimeoutGroup] 2023-04-17 07:59:45,959 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1681718385959,5,FailOnTimeoutGroup] 2023-04-17 07:59:45,959 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:45,960 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-17 07:59:45,963 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:45,963 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:45,964 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 07:59:45,965 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 07:59:45,966 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36 2023-04-17 07:59:45,981 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:39843 2023-04-17 07:59:45,984 INFO [RS:0;jenkins-hbase16:39843] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-17 07:59:45,984 INFO [RS:0;jenkins-hbase16:39843] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-17 07:59:45,984 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1023): About to register with Master. 2023-04-17 07:59:45,986 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,34569,1681718383610 with isa=jenkins-hbase16.apache.org/188.40.62.62:39843, startcode=1681718384756 2023-04-17 07:59:45,999 DEBUG [RS:0;jenkins-hbase16:39843] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-17 07:59:46,041 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44901, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-17 07:59:46,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,063 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36 2023-04-17 07:59:46,063 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46073 2023-04-17 07:59:46,063 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-17 07:59:46,074 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 07:59:46,075 DEBUG [RS:0;jenkins-hbase16:39843] zookeeper.ZKUtil(162): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,075 WARN [RS:0;jenkins-hbase16:39843] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 07:59:46,076 INFO [RS:0;jenkins-hbase16:39843] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 07:59:46,077 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/WALs/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,077 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,39843,1681718384756] 2023-04-17 07:59:46,088 DEBUG [RS:0;jenkins-hbase16:39843] zookeeper.ZKUtil(162): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,099 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-17 07:59:46,105 INFO [RS:0;jenkins-hbase16:39843] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-17 07:59:46,121 INFO [RS:0;jenkins-hbase16:39843] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-17 07:59:46,124 INFO [RS:0;jenkins-hbase16:39843] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-17 07:59:46,124 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,125 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-17 07:59:46,131 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,131 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,131 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,132 DEBUG [RS:0;jenkins-hbase16:39843] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-17 07:59:46,133 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,133 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,133 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,146 INFO [RS:0;jenkins-hbase16:39843] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-17 07:59:46,148 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39843,1681718384756-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:46,161 INFO [RS:0;jenkins-hbase16:39843] regionserver.Replication(203): jenkins-hbase16.apache.org,39843,1681718384756 started 2023-04-17 07:59:46,161 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,39843,1681718384756, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:39843, sessionid=0x10196577f520001 2023-04-17 07:59:46,161 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-17 07:59:46,161 DEBUG [RS:0;jenkins-hbase16:39843] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,161 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,39843,1681718384756' 2023-04-17 07:59:46,161 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:46,162 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:46,162 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-17 07:59:46,162 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-17 07:59:46,163 DEBUG [RS:0;jenkins-hbase16:39843] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,163 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,39843,1681718384756' 2023-04-17 07:59:46,163 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-17 07:59:46,163 DEBUG [RS:0;jenkins-hbase16:39843] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-17 07:59:46,164 DEBUG [RS:0;jenkins-hbase16:39843] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-17 07:59:46,164 INFO [RS:0;jenkins-hbase16:39843] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-17 07:59:46,164 INFO [RS:0;jenkins-hbase16:39843] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-17 07:59:46,282 INFO [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C39843%2C1681718384756, suffix=, logDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/WALs/jenkins-hbase16.apache.org,39843,1681718384756, archiveDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/oldWALs, maxLogs=32 2023-04-17 07:59:46,300 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK] 2023-04-17 07:59:46,304 INFO [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/WALs/jenkins-hbase16.apache.org,39843,1681718384756/jenkins-hbase16.apache.org%2C39843%2C1681718384756.1681718386284 2023-04-17 07:59:46,307 DEBUG [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK]] 2023-04-17 07:59:46,380 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:46,384 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 07:59:46,387 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/info 2023-04-17 07:59:46,388 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 07:59:46,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 07:59:46,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/rep_barrier 2023-04-17 07:59:46,394 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 07:59:46,395 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,396 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 07:59:46,398 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/table 2023-04-17 07:59:46,398 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 07:59:46,399 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,401 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740 2023-04-17 07:59:46,402 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740 2023-04-17 07:59:46,405 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 07:59:46,407 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 07:59:46,411 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 07:59:46,413 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69292674, jitterRate=0.03254130482673645}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 07:59:46,413 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 07:59:46,413 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 07:59:46,413 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 07:59:46,413 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 07:59:46,413 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 07:59:46,413 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 07:59:46,414 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 07:59:46,414 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 07:59:46,420 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 07:59:46,420 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-17 07:59:46,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-17 07:59:46,440 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-17 07:59:46,443 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-17 07:59:46,596 DEBUG [jenkins-hbase16:34569] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-17 07:59:46,601 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,39843,1681718384756, state=OPENING 2023-04-17 07:59:46,615 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-17 07:59:46,624 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:46,625 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 07:59:46,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 07:59:46,818 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:46,821 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-17 07:59:46,825 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-17 07:59:46,838 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-17 07:59:46,838 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 07:59:46,842 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C39843%2C1681718384756.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/WALs/jenkins-hbase16.apache.org,39843,1681718384756, archiveDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/oldWALs, maxLogs=32 2023-04-17 07:59:46,858 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK] 2023-04-17 07:59:46,863 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/WALs/jenkins-hbase16.apache.org,39843,1681718384756/jenkins-hbase16.apache.org%2C39843%2C1681718384756.meta.1681718386844.meta 2023-04-17 07:59:46,864 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42287,DS-d65c3e09-dd61-4a6c-844c-cef43ed99016,DISK]] 2023-04-17 07:59:46,865 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-17 07:59:46,867 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-17 07:59:46,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-17 07:59:46,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-17 07:59:46,895 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-17 07:59:46,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:46,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-17 07:59:46,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-17 07:59:46,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 07:59:46,902 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/info 2023-04-17 07:59:46,902 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/info 2023-04-17 07:59:46,902 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 07:59:46,903 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,904 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 07:59:46,905 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/rep_barrier 2023-04-17 07:59:46,905 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/rep_barrier 2023-04-17 07:59:46,906 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 07:59:46,907 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,907 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 07:59:46,909 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/table 2023-04-17 07:59:46,909 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/table 2023-04-17 07:59:46,909 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 07:59:46,910 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:46,912 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740 2023-04-17 07:59:46,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740 2023-04-17 07:59:46,918 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 07:59:46,920 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 07:59:46,922 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62828970, jitterRate=-0.06377539038658142}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 07:59:46,922 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 07:59:46,931 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681718386809 2023-04-17 07:59:46,946 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-17 07:59:46,947 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-17 07:59:46,948 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,39843,1681718384756, state=OPEN 2023-04-17 07:59:46,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-17 07:59:46,957 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 07:59:46,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-17 07:59:46,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,39843,1681718384756 in 329 msec 2023-04-17 07:59:46,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-17 07:59:46,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 536 msec 2023-04-17 07:59:46,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1220 sec 2023-04-17 07:59:46,975 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681718386975, completionTime=-1 2023-04-17 07:59:46,976 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-17 07:59:46,976 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-17 07:59:47,034 DEBUG [hconnection-0x52b7c078-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:47,039 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:47,060 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-17 07:59:47,060 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681718447060 2023-04-17 07:59:47,061 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681718507061 2023-04-17 07:59:47,062 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-04-17 07:59:47,101 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34569,1681718383610-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:47,102 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34569,1681718383610-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:47,102 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34569,1681718383610-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:47,103 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:34569, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:47,103 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-17 07:59:47,110 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-17 07:59:47,119 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-17 07:59:47,120 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 07:59:47,129 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-17 07:59:47,131 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 07:59:47,134 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 07:59:47,155 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:47,157 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d empty. 2023-04-17 07:59:47,158 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:47,159 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-17 07:59:47,587 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-17 07:59:47,592 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c90e072c3dc123157787d896ec6fad3d, NAME => 'hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 07:59:47,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:47,612 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c90e072c3dc123157787d896ec6fad3d, disabling compactions & flushes 2023-04-17 07:59:47,612 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:47,612 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:47,612 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. after waiting 0 ms 2023-04-17 07:59:47,612 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:47,612 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:47,612 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c90e072c3dc123157787d896ec6fad3d: 2023-04-17 07:59:47,618 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 07:59:47,634 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681718387622"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718387622"}]},"ts":"1681718387622"} 2023-04-17 07:59:47,656 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 07:59:47,658 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 07:59:47,664 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718387659"}]},"ts":"1681718387659"} 2023-04-17 07:59:47,668 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-17 07:59:47,740 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c90e072c3dc123157787d896ec6fad3d, ASSIGN}] 2023-04-17 07:59:47,743 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c90e072c3dc123157787d896ec6fad3d, ASSIGN 2023-04-17 07:59:47,745 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c90e072c3dc123157787d896ec6fad3d, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 07:59:47,898 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c90e072c3dc123157787d896ec6fad3d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:47,899 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681718387897"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718387897"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718387897"}]},"ts":"1681718387897"} 2023-04-17 07:59:47,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c90e072c3dc123157787d896ec6fad3d, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 07:59:48,070 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:48,071 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c90e072c3dc123157787d896ec6fad3d, NAME => 'hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.', STARTKEY => '', ENDKEY => ''} 2023-04-17 07:59:48,072 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,073 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:48,073 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,073 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,076 INFO [StoreOpener-c90e072c3dc123157787d896ec6fad3d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,079 DEBUG [StoreOpener-c90e072c3dc123157787d896ec6fad3d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/info 2023-04-17 07:59:48,079 DEBUG [StoreOpener-c90e072c3dc123157787d896ec6fad3d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/info 2023-04-17 07:59:48,080 INFO [StoreOpener-c90e072c3dc123157787d896ec6fad3d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c90e072c3dc123157787d896ec6fad3d columnFamilyName info 2023-04-17 07:59:48,081 INFO [StoreOpener-c90e072c3dc123157787d896ec6fad3d-1] regionserver.HStore(310): Store=c90e072c3dc123157787d896ec6fad3d/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:48,082 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,083 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,087 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for c90e072c3dc123157787d896ec6fad3d 2023-04-17 07:59:48,090 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 07:59:48,091 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened c90e072c3dc123157787d896ec6fad3d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63496673, jitterRate=-0.05382584035396576}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 07:59:48,091 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for c90e072c3dc123157787d896ec6fad3d: 2023-04-17 07:59:48,093 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d., pid=6, masterSystemTime=1681718388062 2023-04-17 07:59:48,097 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:48,098 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 07:59:48,099 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c90e072c3dc123157787d896ec6fad3d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:48,099 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681718388098"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718388098"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718388098"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718388098"}]},"ts":"1681718388098"} 2023-04-17 07:59:48,106 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-17 07:59:48,106 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c90e072c3dc123157787d896ec6fad3d, server=jenkins-hbase16.apache.org,39843,1681718384756 in 194 msec 2023-04-17 07:59:48,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-17 07:59:48,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c90e072c3dc123157787d896ec6fad3d, ASSIGN in 366 msec 2023-04-17 07:59:48,111 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 07:59:48,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718388112"}]},"ts":"1681718388112"} 2023-04-17 07:59:48,115 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-17 07:59:48,128 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 07:59:48,131 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.0070 sec 2023-04-17 07:59:48,132 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-17 07:59:48,140 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-17 07:59:48,141 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 07:59:48,180 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-17 07:59:48,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 07:59:48,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 46 msec 2023-04-17 07:59:48,225 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-17 07:59:48,249 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 07:59:48,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 38 msec 2023-04-17 07:59:48,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-17 07:59:48,299 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-17 07:59:48,300 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.436sec 2023-04-17 07:59:48,302 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-17 07:59:48,303 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-17 07:59:48,303 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-17 07:59:48,304 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34569,1681718383610-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-17 07:59:48,305 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34569,1681718383610-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-17 07:59:48,315 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-17 07:59:48,318 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x7ddc537c to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:48,330 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1545112e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:48,347 DEBUG [hconnection-0x7019db20-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:48,357 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:48,367 INFO [Listener at localhost.localdomain/40407] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 07:59:48,384 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=169, AvailableMemoryMB=8606 2023-04-17 07:59:48,403 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 07:59:48,407 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 07:59:48,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 07:59:48,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 07:59:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 07:59:48,427 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 07:59:48,430 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 07:59:48,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-17 07:59:48,434 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:48,435 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a empty. 2023-04-17 07:59:48,437 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:48,437 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 07:59:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:48,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:48,858 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 07:59:48,863 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2a1678219a4c91eb2f28c4b713537f8a, NAME => 'TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 07:59:49,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:49,299 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:49,300 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2a1678219a4c91eb2f28c4b713537f8a, disabling compactions & flushes 2023-04-17 07:59:49,300 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,301 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,301 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. after waiting 0 ms 2023-04-17 07:59:49,301 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,301 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,301 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:49,308 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 07:59:49,310 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718389310"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718389310"}]},"ts":"1681718389310"} 2023-04-17 07:59:49,313 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 07:59:49,315 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 07:59:49,315 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718389315"}]},"ts":"1681718389315"} 2023-04-17 07:59:49,317 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 07:59:49,363 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, ASSIGN}] 2023-04-17 07:59:49,366 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, ASSIGN 2023-04-17 07:59:49,369 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 07:59:49,522 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2a1678219a4c91eb2f28c4b713537f8a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:49,522 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718389522"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718389522"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718389522"}]},"ts":"1681718389522"} 2023-04-17 07:59:49,528 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 07:59:49,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:49,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2a1678219a4c91eb2f28c4b713537f8a, NAME => 'TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.', STARTKEY => '', ENDKEY => ''} 2023-04-17 07:59:49,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 07:59:49,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,694 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,696 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A 2023-04-17 07:59:49,696 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A 2023-04-17 07:59:49,698 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 07:59:49,698 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a1678219a4c91eb2f28c4b713537f8a columnFamilyName A 2023-04-17 07:59:49,699 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(310): Store=2a1678219a4c91eb2f28c4b713537f8a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:49,699 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,701 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B 2023-04-17 07:59:49,701 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B 2023-04-17 07:59:49,701 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 07:59:49,702 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a1678219a4c91eb2f28c4b713537f8a columnFamilyName B 2023-04-17 07:59:49,703 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(310): Store=2a1678219a4c91eb2f28c4b713537f8a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:49,703 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,705 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C 2023-04-17 07:59:49,705 DEBUG [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C 2023-04-17 07:59:49,705 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 07:59:49,706 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a1678219a4c91eb2f28c4b713537f8a columnFamilyName C 2023-04-17 07:59:49,706 INFO [StoreOpener-2a1678219a4c91eb2f28c4b713537f8a-1] regionserver.HStore(310): Store=2a1678219a4c91eb2f28c4b713537f8a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 07:59:49,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,713 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 07:59:49,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:49,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 07:59:49,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 2a1678219a4c91eb2f28c4b713537f8a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63943490, jitterRate=-0.04716774821281433}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 07:59:49,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:49,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., pid=11, masterSystemTime=1681718389683 2023-04-17 07:59:49,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:49,724 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2a1678219a4c91eb2f28c4b713537f8a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:49,725 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718389724"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718389724"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718389724"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718389724"}]},"ts":"1681718389724"} 2023-04-17 07:59:49,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-17 07:59:49,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 in 199 msec 2023-04-17 07:59:49,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-17 07:59:49,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, ASSIGN in 367 msec 2023-04-17 07:59:49,734 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 07:59:49,734 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718389734"}]},"ts":"1681718389734"} 2023-04-17 07:59:49,736 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 07:59:49,747 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 07:59:49,749 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.3250 sec 2023-04-17 07:59:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 07:59:50,561 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-17 07:59:50,565 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x490ff358 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,605 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ad5f945, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,608 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,611 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,614 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 07:59:50,617 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:36050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 07:59:50,626 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,638 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142c8aea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,639 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x45864842 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,651 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,651 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,666 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f537ba4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,667 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x66ca212c to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,681 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b5e40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,681 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x1ecabb06 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,692 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@127a87ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,693 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,705 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,706 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3e044e22 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,751 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a7ef79d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,753 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x6ab9467e to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,767 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@776686ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,768 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x18f7ec13 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 07:59:50,784 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@385dd3c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 07:59:50,793 DEBUG [hconnection-0x1a108d79-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,795 DEBUG [hconnection-0x2ea49fc0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,795 DEBUG [hconnection-0x7e6ea1cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,796 DEBUG [hconnection-0x445d2556-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,799 DEBUG [hconnection-0x339f0d0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,799 DEBUG [hconnection-0x35e9f75d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,800 DEBUG [hconnection-0x3479d5e5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,800 DEBUG [hconnection-0x126c60ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,800 DEBUG [hconnection-0x4f593eba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 07:59:50,802 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,804 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,807 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,810 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,810 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,811 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,813 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,813 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,814 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 07:59:50,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:50,846 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:50,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:50,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:50,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:50,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:50,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:50,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:50,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:50,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:50,859 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,861 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:50,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:50,866 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:50,866 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:50,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:50,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:50,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 07:59:50,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:50,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:50,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:50,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:50,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:50,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:50,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:50,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:50,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:50,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:50,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:50,907 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:50,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:50,908 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,908 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:50,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:50,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:50,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:50,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:50,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:50,910 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:50,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aaf50ba[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:50,910 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,921 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:50,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:50,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:50,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:50,936 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:50,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:50,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:50,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:50,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:50,944 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:50,944 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:50,944 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:50,944 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:50,944 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:50,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:50,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:50,964 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:50,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:50,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:50,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:50,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:50,991 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:50,991 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:50,991 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:50,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718450993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718451000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718451003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718390960, max timestamp=1681718390998Num uniques 859; 2023-04-17 07:59:51,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718390960, max timestamp=1681718390998Num uniques 859; 2023-04-17 07:59:51,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:51,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718390960, max timestamp=1681718390995Num uniques 859; 2023-04-17 07:59:51,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 07:59:51,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:51,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/6f1f6e9540184054857e1ac14317eb7b 2023-04-17 07:59:51,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:51,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:51,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718451117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718451119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:51,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:51,127 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:51,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:51,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:51,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:51,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:51,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:51,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:51,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,164 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:51,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:51,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:51,165 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,165 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:51,165 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:51,165 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:51,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:51,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:51,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:51,174 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,175 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,176 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,176 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:51,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718451172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718451172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718451178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f544cf[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:51,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:51,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:51,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:51,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:51,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:51,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:51,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,200 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:51,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:51,217 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:51,217 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:51,217 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:51,217 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:51,217 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:51,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718451223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718451230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/625341aeb25f448f960d666705da3621 2023-04-17 07:59:51,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:51,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,275 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:51,274 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,275 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,275 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,275 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:51,275 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:51,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/6a5966b5caba49999602c0f9f1a75690 2023-04-17 07:59:51,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/6f1f6e9540184054857e1ac14317eb7b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b 2023-04-17 07:59:51,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 07:59:51,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/625341aeb25f448f960d666705da3621 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621 2023-04-17 07:59:51,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 07:59:51,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/6a5966b5caba49999602c0f9f1a75690 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690 2023-04-17 07:59:51,375 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:51,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 07:59:51,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.08 KB/130125 for 2a1678219a4c91eb2f28c4b713537f8a in 501ms, sequenceid=13, compaction requested=false 2023-04-17 07:59:51,385 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 07:59:51,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:51,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:51,394 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:51,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:51,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:51,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:51,406 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-17 07:59:51,406 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:51,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:51,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:51,407 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:51,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:51,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:51,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:51,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:51,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:51,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:51,419 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:51,419 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,424 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:51,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:51,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:51,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:51,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:51,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:51,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:51,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:51,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:51,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:51,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c854a5[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:51,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:51,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,466 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,467 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:51,467 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:51,467 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:51,467 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,467 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:51,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718451449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,482 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,486 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,486 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/a8721e52aead42fe99ca3649a63dcae3 2023-04-17 07:59:51,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718451468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718451474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718451479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:51,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:51,495 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:51,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:51,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:51,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:51,495 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:51,510 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,510 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,510 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,517 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718451489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,540 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,540 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,541 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,541 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,541 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,543 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,543 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,543 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,543 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,543 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:51,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:51,550 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:51,550 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:51,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718451583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718451594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718451597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718451597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718451625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:51,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:51,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:51,684 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:51,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,686 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:51,686 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:51,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:51,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:51,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:51,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:51,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:51,700 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:51,700 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:51,700 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,700 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:51,700 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:51,700 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,701 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:51,701 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:51,701 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,701 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:51,701 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:51,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,717 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,717 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,717 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f81686[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:51,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:51,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,733 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:51,733 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:51,733 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:51,734 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,734 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:51,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:51,742 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,742 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:51,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:51,746 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:51,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:51,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:51,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:51,746 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:51,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:51,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,783 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,783 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,783 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:51,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:51,783 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:51,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:51,789 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:51,789 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:51,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718451797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718451808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718451832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718451833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:51,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718451854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,889 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:51,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:51,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:51,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:51,914 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:51,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:51,923 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:51,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:51,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:51,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:51,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:51,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:51,939 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:51,939 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:51,944 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:51,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:51,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:51,956 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2c091b60c88c40a78605be0f48c2a126 2023-04-17 07:59:51,964 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,964 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:51,964 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:51,964 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:51,969 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:51,969 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:51,969 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,983 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:51,984 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:51,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:51,999 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:52,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f2be63[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:52,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:52,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,016 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,019 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:52,019 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:52,019 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:52,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:52,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:52,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:52,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:52,033 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:52,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:52,034 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:52,034 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:52,034 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:52,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,043 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,080 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:52,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:52,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718452117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718452128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718452150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718452155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,180 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:52,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:52,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:52,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718452189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:52,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:52,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:52,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:52,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:52,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:52,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:52,218 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:52,218 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:52,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:52,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:52,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:52,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:52,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:52,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:52,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:52,225 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:52,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:52,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8a574e[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:52,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:52,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:52,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:52,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:52,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:52,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,258 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:52,258 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,258 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:52,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:52,266 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:52,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:52,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:52,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:52,267 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:52,276 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,276 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,276 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:52,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,310 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,310 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:52,310 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:52,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:52,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:52,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:52,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:52,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:52,431 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:52,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:52,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:52,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,441 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,441 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:52,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:52,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:52,443 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:52,443 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,444 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:52,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:52,447 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:52,447 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,447 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:52,447 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:52,447 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:52,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:52,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:52,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:52,457 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e4a0de06c24a46b083fddf5d18cffe84 2023-04-17 07:59:52,461 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,461 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:52,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a705526[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:52,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:52,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:52,485 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,485 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:52,485 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:52,485 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:52,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/a8721e52aead42fe99ca3649a63dcae3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3 2023-04-17 07:59:52,504 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 07:59:52,505 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:52,505 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,505 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:52,529 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3, entries=200, sequenceid=50, filesize=13.9 K 2023-04-17 07:59:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2c091b60c88c40a78605be0f48c2a126 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126 2023-04-17 07:59:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:52,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:52,558 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:52,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:52,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:52,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:52,558 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:52,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,575 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,589 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,589 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:52,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:52,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:52,591 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:52,591 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:52,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 07:59:52,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e4a0de06c24a46b083fddf5d18cffe84 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84 2023-04-17 07:59:52,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 07:59:52,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=53.67 KB/54960 for 2a1678219a4c91eb2f28c4b713537f8a in 1232ms, sequenceid=50, compaction requested=false 2023-04-17 07:59:52,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:52,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:52,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:52,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:52,692 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:52,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:52,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:52,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:52,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:52,705 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:52,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:52,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:52,711 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,711 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,714 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-17 07:59:52,715 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-17 07:59:52,722 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/ed83bcbf5f634089a97099de8f5cfaaf 2023-04-17 07:59:52,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:52,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:52,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:52,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:52,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,725 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:52,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718452713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,727 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:52,727 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:52,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718452715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:52,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:52,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:52,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:52,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,733 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:52,733 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:52,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718452726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718452726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718452727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:52,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cea3ec8[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:52,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:52,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:52,758 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:52,758 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:52,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:52,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,766 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:52,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:52,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:52,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:52,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:52,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:52,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:52,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:52,774 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,774 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,774 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,775 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:52,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:52,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:52,809 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:52,809 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:52,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,816 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,817 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,817 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,817 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,817 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/1ba39d27fad3418b98b5cc23eeb99e90 2023-04-17 07:59:52,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:52,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718452829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718452830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718452838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718452845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718452860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e8fb8f0bb97b4b77acadb2a03bf3d732 2023-04-17 07:59:52,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/ed83bcbf5f634089a97099de8f5cfaaf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf 2023-04-17 07:59:52,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf, entries=150, sequenceid=63, filesize=11.6 K 2023-04-17 07:59:52,909 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:52,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/1ba39d27fad3418b98b5cc23eeb99e90 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90 2023-04-17 07:59:52,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:52,922 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:52,923 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:52,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:52,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90, entries=150, sequenceid=63, filesize=11.6 K 2023-04-17 07:59:52,943 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:52,943 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e8fb8f0bb97b4b77acadb2a03bf3d732 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732 2023-04-17 07:59:52,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:52,944 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,944 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:52,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:52,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:52,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:52,949 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:52,949 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:52,952 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:52,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:52,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:52,956 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,956 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:52,956 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:52,956 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:52,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:52,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:52,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,959 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:52,959 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:52,965 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,965 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,966 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,966 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:52,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732, entries=150, sequenceid=63, filesize=11.6 K 2023-04-17 07:59:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2a1678219a4c91eb2f28c4b713537f8a in 325ms, sequenceid=63, compaction requested=true 2023-04-17 07:59:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:52,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f488290[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:52,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:52,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,982 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:52,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:52,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:52,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:52,983 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,983 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:52,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:52,984 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:52,988 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:52,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:52,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:52,993 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:52,993 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,993 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,993 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:52,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:52,998 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:52,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:52,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:52,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:52,999 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:53,002 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:53,004 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 07:59:53,004 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,004 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=34.8 K 2023-04-17 07:59:53,005 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 625341aeb25f448f960d666705da3621, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718390835 2023-04-17 07:59:53,006 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:53,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,007 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,007 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,006 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2c091b60c88c40a78605be0f48c2a126, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681718391101 2023-04-17 07:59:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,017 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 07:59:53,017 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,017 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=37.1 K 2023-04-17 07:59:53,017 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1ba39d27fad3418b98b5cc23eeb99e90, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:53,016 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,018 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6f1f6e9540184054857e1ac14317eb7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718390835 2023-04-17 07:59:53,018 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,022 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a8721e52aead42fe99ca3649a63dcae3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681718391101 2023-04-17 07:59:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,023 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ed83bcbf5f634089a97099de8f5cfaaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,040 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:53,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:53,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:53,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:53,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,047 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:53,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:53,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:53,060 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#9 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:53,087 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#11 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:53,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718453096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718453102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718453111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718453103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/340e457151664f4b8d9f5e9878533874 2023-04-17 07:59:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718453117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,145 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:53,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:53,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:53,160 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:53,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:53,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:53,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,174 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,175 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,175 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:53,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:53,183 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/ada35bd9d03f4ebb9904198f8609f6b5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ada35bd9d03f4ebb9904198f8609f6b5 2023-04-17 07:59:53,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:53,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:53,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:53,194 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,195 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:53,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:53,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:53,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:53,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:53,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:53,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:53,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:53,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:53,215 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,216 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,216 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:53,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b79a10[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:53,219 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:53,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,224 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into ada35bd9d03f4ebb9904198f8609f6b5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:53,224 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:53,224 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718392975; duration=0sec 2023-04-17 07:59:53,225 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:53,225 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:53,228 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:53,228 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 07:59:53,228 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,228 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=34.8 K 2023-04-17 07:59:53,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:53,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:53,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:53,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:53,232 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6a5966b5caba49999602c0f9f1a75690, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718390835 2023-04-17 07:59:53,233 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e4a0de06c24a46b083fddf5d18cffe84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681718391101 2023-04-17 07:59:53,235 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e8fb8f0bb97b4b77acadb2a03bf3d732, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:53,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718453229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718453231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718453233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:53,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:53,240 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718453233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718453244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:53,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:53,250 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:53,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:53,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:53,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:53,251 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:53,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,258 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,259 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,270 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#13 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:53,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:53,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:53,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:53,300 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/29303d82939940cfbef0f0cc05093363 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/29303d82939940cfbef0f0cc05093363 2023-04-17 07:59:53,313 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 29303d82939940cfbef0f0cc05093363(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:53,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:53,314 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718392988; duration=0sec 2023-04-17 07:59:53,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:53,372 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:53,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:53,382 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:53,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:53,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:53,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:53,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:53,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:53,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:53,394 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:53,394 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:53,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:53,404 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,404 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:53,404 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:53,404 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:53,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:53,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:53,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:53,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:53,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:53,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e63045[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:53,419 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:53,419 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:53,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:53,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:53,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:53,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:53,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:53,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:53,443 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:53,443 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:53,443 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:53,443 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:53,443 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:53,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718453443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718453444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718453446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718453452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,458 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718453453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:53,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,473 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:53,473 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,526 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/ff007bdc7f584c4e89c92f897d211311 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ff007bdc7f584c4e89c92f897d211311 2023-04-17 07:59:53,539 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into ff007bdc7f584c4e89c92f897d211311(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:53,539 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:53,539 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718392984; duration=0sec 2023-04-17 07:59:53,540 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:53,573 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:53,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:53,582 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:53,582 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:53,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:53,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/f9c32efaa0ef4149a2cfae714d6c2ac3 2023-04-17 07:59:53,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:53,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:53,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:53,617 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:53,617 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,618 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:53,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:53,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:53,630 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,630 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:53,630 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:53,630 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:53,641 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:53,641 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,641 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:53,641 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:53,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:53,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:53,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357755ef[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:53,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:53,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,668 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,668 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:53,668 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:53,668 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:53,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:53,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:53,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:53,681 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:53,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:53,694 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:53,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:53,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:53,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:53,694 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:53,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/c0f97477f09e4d08985191c597ce8989 2023-04-17 07:59:53,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:53,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/340e457151664f4b8d9f5e9878533874 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:53,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:53,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:53,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:53,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:53,716 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:53,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874, entries=200, sequenceid=83, filesize=13.9 K 2023-04-17 07:59:53,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:53,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:53,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/f9c32efaa0ef4149a2cfae714d6c2ac3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3 2023-04-17 07:59:53,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718453749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718453749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718453753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718453755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3, entries=150, sequenceid=83, filesize=11.6 K 2023-04-17 07:59:53,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/c0f97477f09e4d08985191c597ce8989 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989 2023-04-17 07:59:53,767 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 07:59:53,773 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-17 07:59:53,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:53,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718453763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989, entries=150, sequenceid=83, filesize=11.6 K 2023-04-17 07:59:53,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2a1678219a4c91eb2f28c4b713537f8a in 756ms, sequenceid=83, compaction requested=false 2023-04-17 07:59:53,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:53,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:53,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:53,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:53,823 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:53,823 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,825 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:53,825 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:53,826 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,826 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,837 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:53,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:53,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:53,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:53,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:53,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:53,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:53,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:53,840 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:53,844 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:53,844 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:53,850 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:53,851 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:53,851 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 07:59:53,851 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:53,853 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:53,853 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:53,853 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:53,853 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:53,853 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:54,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:54,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:54,291 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/0075b27d356a45ae9859acf4689f6689 2023-04-17 07:59:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718454289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718454290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718454290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718454295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58356 deadline: 1681718454299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,349 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2d4345c17c7148a6b522cbe4f78acb70 2023-04-17 07:59:54,376 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/b4b9404e9a154fc99d7d71d1719813c3 2023-04-17 07:59:54,386 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/0075b27d356a45ae9859acf4689f6689 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689 2023-04-17 07:59:54,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718454396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718454396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718454396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718454401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,407 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689, entries=150, sequenceid=98, filesize=11.6 K 2023-04-17 07:59:54,410 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2d4345c17c7148a6b522cbe4f78acb70 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70 2023-04-17 07:59:54,422 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70, entries=150, sequenceid=98, filesize=11.6 K 2023-04-17 07:59:54,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/b4b9404e9a154fc99d7d71d1719813c3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3 2023-04-17 07:59:54,442 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3, entries=150, sequenceid=98, filesize=11.6 K 2023-04-17 07:59:54,444 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2a1678219a4c91eb2f28c4b713537f8a in 593ms, sequenceid=98, compaction requested=true 2023-04-17 07:59:54,444 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:54,444 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:54,444 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:54,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:54,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:54,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e101dd3[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:54,485 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:54,485 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:54,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:54,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:54,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:54,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:54,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:54,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:54,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:54,509 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:54,510 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:54,510 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:54,510 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:54,510 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:54,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,522 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:54,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:54,533 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,533 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:54,534 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:54,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394276, max timestamp=1681718394600Num uniques 859; 2023-04-17 07:59:54,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394276, max timestamp=1681718394600Num uniques 859; 2023-04-17 07:59:54,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:54,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=64.85 KB heapSize=184.09 KB 2023-04-17 07:59:54,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394276, max timestamp=1681718394600Num uniques 859; 2023-04-17 07:59:54,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:54,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:54,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:54,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:54,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:54,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:54,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:54,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:54,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:54,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:54,658 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:54,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:54,659 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:54,659 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:54,659 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,659 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:54,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:54,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:54,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394657, max timestamp=1681718394666Num uniques 859; 2023-04-17 07:59:54,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:54,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:54,670 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:54,670 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:54,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:54,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394657, max timestamp=1681718394668Num uniques 859; 2023-04-17 07:59:54,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:54,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:54,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:54,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:54,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:54,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:54,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:54,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:54,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:54,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:54,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718394657, max timestamp=1681718394666Num uniques 859; 2023-04-17 07:59:54,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 07:59:54,682 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,682 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,683 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,683 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:54,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b711b[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:54,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:54,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:54,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:54,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:54,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:54,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:54,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:54,709 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:54,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:54,713 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:54,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:54,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:54,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:54,714 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:54,724 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,724 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,724 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681718394707, max timestamp=1681718394718Num uniques 1018; 2023-04-17 07:59:54,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 07:59:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718394707, max timestamp=1681718394718Num uniques 1018; 2023-04-17 07:59:54,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 07:59:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681718394707, max timestamp=1681718394718Num uniques 1018; 2023-04-17 07:59:54,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 07:59:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,741 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:54,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,742 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:54,742 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,742 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:54,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718454765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718394751, max timestamp=1681718394767Num uniques 1077; 2023-04-17 07:59:54,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 07:59:54,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718394751, max timestamp=1681718394767Num uniques 1077; 2023-04-17 07:59:54,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 07:59:54,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718394751, max timestamp=1681718394767Num uniques 1027; 2023-04-17 07:59:54,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-17 07:59:54,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718454811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681718394795, max timestamp=1681718394811Num uniques 1086; 2023-04-17 07:59:54,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681718394795, max timestamp=1681718394811Num uniques 1086; 2023-04-17 07:59:54,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-17 07:59:54,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-17 07:59:54,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718394795, max timestamp=1681718394811Num uniques 1086; 2023-04-17 07:59:54,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-04-17 07:59:54,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:54,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:54,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:54,848 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:54,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:54,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:54,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:54,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:54,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,861 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,861 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:54,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:54,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:54,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:54,876 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:54,876 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:54,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:54,882 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:54,882 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:54,884 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:54,884 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:54,884 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:54,884 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:54,884 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:54,884 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:54,884 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,884 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:54,884 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:54,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718454873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718454873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718454884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,891 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,891 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:54,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b161d09[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:54,893 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:54,893 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,916 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,916 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:54,916 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:54,916 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:54,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718454919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:54,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:54,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:54,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:54,930 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:54,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:54,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:54,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:54,930 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:54,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:54,941 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,941 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,954 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,966 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:54,966 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:54,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:54,966 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:54,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:54,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:54,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:54,966 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,967 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:54,967 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,967 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,967 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:54,967 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:54,967 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:54,967 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:54,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:54,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718454989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:54,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:54,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718454990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/da5821a905ef4e0c943ab157da637e50 2023-04-17 07:59:55,067 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:55,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:55,080 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:55,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:55,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:55,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718455088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:55,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:55,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:55,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:55,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:55,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/6114fca8d72a4bffb2c83b434053e324 2023-04-17 07:59:55,107 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,107 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:55,107 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:55,110 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:55,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:55,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:55,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:55,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:55,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1f9a9[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:55,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:55,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:55,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58330 deadline: 1681718455123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,133 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,134 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:55,134 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:55,134 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:55,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:55,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/d87a01c5dc0942ebaabc1a989b590fa8 2023-04-17 07:59:55,146 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,146 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:55,146 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,146 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/da5821a905ef4e0c943ab157da637e50 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50 2023-04-17 07:59:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:55,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:55,157 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:55,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:55,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:55,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:55,157 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:55,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50, entries=150, sequenceid=121, filesize=11.6 K 2023-04-17 07:59:55,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/6114fca8d72a4bffb2c83b434053e324 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324 2023-04-17 07:59:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,169 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324, entries=150, sequenceid=121, filesize=11.6 K 2023-04-17 07:59:55,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/d87a01c5dc0942ebaabc1a989b590fa8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8 2023-04-17 07:59:55,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:55,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,183 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:55,184 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:55,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58318 deadline: 1681718455196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58340 deadline: 1681718455196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8, entries=150, sequenceid=121, filesize=11.6 K 2023-04-17 07:59:55,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=150.46 KB/154074 for 2a1678219a4c91eb2f28c4b713537f8a in 603ms, sequenceid=121, compaction requested=true 2023-04-17 07:59:55,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:55,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:55,210 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 07:59:55,210 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 07:59:55,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:55,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:55,212 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 07:59:55,212 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 07:59:55,212 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,212 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ff007bdc7f584c4e89c92f897d211311, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=46.5 K 2023-04-17 07:59:55,213 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49987 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 07:59:55,213 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 07:59:55,213 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,213 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ada35bd9d03f4ebb9904198f8609f6b5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.8 K 2023-04-17 07:59:55,214 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ada35bd9d03f4ebb9904198f8609f6b5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:55,214 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ff007bdc7f584c4e89c92f897d211311, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:55,215 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 340e457151664f4b8d9f5e9878533874, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681718392704 2023-04-17 07:59:55,216 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f9c32efaa0ef4149a2cfae714d6c2ac3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681718392704 2023-04-17 07:59:55,216 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 0075b27d356a45ae9859acf4689f6689, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681718393071 2023-04-17 07:59:55,217 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting da5821a905ef4e0c943ab157da637e50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:55,217 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2d4345c17c7148a6b522cbe4f78acb70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681718393071 2023-04-17 07:59:55,218 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6114fca8d72a4bffb2c83b434053e324, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:55,241 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:55,243 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:55,284 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:55,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:55,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:55,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:55,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:55,296 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:55,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:55,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:55,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,304 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/ec7f9684da614d219e5153cd1e4d49e4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ec7f9684da614d219e5153cd1e4d49e4 2023-04-17 07:59:55,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:55,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:55,316 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into ec7f9684da614d219e5153cd1e4d49e4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:55,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:55,316 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=12, startTime=1681718395210; duration=0sec 2023-04-17 07:59:55,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:55,317 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 07:59:55,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:55,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:55,318 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:55,318 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:55,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718394858, max timestamp=1681718395314Num uniques 1245; 2023-04-17 07:59:55,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=8930, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-17 07:59:55,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=110.21 KB heapSize=306.33 KB 2023-04-17 07:59:55,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:55,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718394858, max timestamp=1681718395314Num uniques 1245; 2023-04-17 07:59:55,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=8930, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-17 07:59:55,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:55,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:55,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:55,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:55,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718394858, max timestamp=1681718395314Num uniques 1009; 2023-04-17 07:59:55,321 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:55,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-04-17 07:59:55,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:55,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:55,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:55,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,324 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 07:59:55,324 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 07:59:55,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:55,324 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:55,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,324 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/29303d82939940cfbef0f0cc05093363, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=46.5 K 2023-04-17 07:59:55,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:55,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:55,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:55,325 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:55,326 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 29303d82939940cfbef0f0cc05093363, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681718391448 2023-04-17 07:59:55,328 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c0f97477f09e4d08985191c597ce8989, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681718392704 2023-04-17 07:59:55,328 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b4b9404e9a154fc99d7d71d1719813c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681718393071 2023-04-17 07:59:55,329 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d87a01c5dc0942ebaabc1a989b590fa8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:55,333 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,333 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,333 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,333 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:55,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a319267[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:55,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:55,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:55,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:55,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:55,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:55,357 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:55,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,358 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:55,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:55,361 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:55,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:55,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:55,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:55,361 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:55,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/3d01738eabfd4d8a9f24537db00ae79b 2023-04-17 07:59:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,416 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:55,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:55,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:55,421 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/c6cff3b039b64056b20612a1f4597f0b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c6cff3b039b64056b20612a1f4597f0b 2023-04-17 07:59:55,434 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into c6cff3b039b64056b20612a1f4597f0b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:55,434 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:55,434 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=12, startTime=1681718395210; duration=0sec 2023-04-17 07:59:55,434 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:55,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718395424, max timestamp=1681718395437Num uniques 859; 2023-04-17 07:59:55,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718395424, max timestamp=1681718395437Num uniques 859; 2023-04-17 07:59:55,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 07:59:55,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718395424, max timestamp=1681718395437Num uniques 859; 2023-04-17 07:59:55,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 07:59:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 07:59:55,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718395463, max timestamp=1681718395478Num uniques 1118; 2023-04-17 07:59:55,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-17 07:59:55,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718395463, max timestamp=1681718395478Num uniques 1118; 2023-04-17 07:59:55,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718395463, max timestamp=1681718395478Num uniques 1118; 2023-04-17 07:59:55,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-17 07:59:55,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-17 07:59:55,517 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:55,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8096, totalHeapSize=18648, min timestamp=1681718395507, max timestamp=1681718395516Num uniques 1127; 2023-04-17 07:59:55,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8098, totalHeapSize=18648, min timestamp=1681718395507, max timestamp=1681718395516Num uniques 1127; 2023-04-17 07:59:55,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=51606, new segment data size=8096, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-17 07:59:55,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51608, new segment data size=8098, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-17 07:59:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718395507, max timestamp=1681718395515Num uniques 1127; 2023-04-17 07:59:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-04-17 07:59:55,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:55,525 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:55,525 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:55,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:55,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:55,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:55,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:55,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:55,553 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:55,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:55,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:55,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:55,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:55,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:55,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:55,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:55,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718395549, max timestamp=1681718395561Num uniques 1036; 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718395549, max timestamp=1681718395561Num uniques 1036; 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718395549, max timestamp=1681718395561Num uniques 1009; 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-17 07:59:55,565 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,566 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,566 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:55,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-17 07:59:55,565 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:55,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539bd993[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:55,568 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:55,568 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:55,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:55,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:55,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:55,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681718395579, max timestamp=1681718395603Num uniques 1095; 2023-04-17 07:59:55,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-17 07:59:55,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718395589, max timestamp=1681718395603Num uniques 1095; 2023-04-17 07:59:55,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 07:59:55,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718395589, max timestamp=1681718395603Num uniques 1095; 2023-04-17 07:59:55,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 07:59:55,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:55,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:55,628 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:55,628 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:55,628 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:55,628 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:55,628 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:55,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,646 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718395641, max timestamp=1681718395649Num uniques 1204; 2023-04-17 07:59:55,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-17 07:59:55,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718395641, max timestamp=1681718395649Num uniques 1054; 2023-04-17 07:59:55,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-17 07:59:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718395641, max timestamp=1681718395649Num uniques 1054; 2023-04-17 07:59:55,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-17 07:59:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:55,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:55,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:55,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,667 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,667 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:55,667 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,668 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:55,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681718395660, max timestamp=1681718395680Num uniques 1063; 2023-04-17 07:59:55,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-17 07:59:55,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718395672, max timestamp=1681718395680Num uniques 1063; 2023-04-17 07:59:55,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-17 07:59:55,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718395672, max timestamp=1681718395680Num uniques 1063; 2023-04-17 07:59:55,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-17 07:59:55,688 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/0f0f12938d724529b09edbaec2446f65 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0f0f12938d724529b09edbaec2446f65 2023-04-17 07:59:55,712 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 0f0f12938d724529b09edbaec2446f65(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:55,713 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:55,713 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=12, startTime=1681718395209; duration=0sec 2023-04-17 07:59:55,713 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:55,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681718395703, max timestamp=1681718395718Num uniques 1072; 2023-04-17 07:59:55,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-17 07:59:55,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681718395703, max timestamp=1681718395718Num uniques 1122; 2023-04-17 07:59:55,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-17 07:59:55,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681718395703, max timestamp=1681718395718Num uniques 1072; 2023-04-17 07:59:55,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718395740, max timestamp=1681718395743Num uniques 1131; 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718395740, max timestamp=1681718395743Num uniques 1131; 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718395740, max timestamp=1681718395743Num uniques 1131; 2023-04-17 07:59:55,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-17 07:59:55,768 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:55,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:55,772 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:55,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:55,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:55,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1681718395760, max timestamp=1681718395774Num uniques 1040; 2023-04-17 07:59:55,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=10980, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-17 07:59:55,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1681718395760, max timestamp=1681718395774Num uniques 1040; 2023-04-17 07:59:55,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=10980, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-17 07:59:55,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718395760, max timestamp=1681718395774Num uniques 1040; 2023-04-17 07:59:55,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-04-17 07:59:55,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:55,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:55,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:55,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:55,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:55,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:55,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:55,798 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,798 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:55,798 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:55,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:55,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:55,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:55,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:55,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:55,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:55,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15977, totalHeapSize=36536, min timestamp=1681718395795, max timestamp=1681718395807Num uniques 1099; 2023-04-17 07:59:55,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50327, new segment data size=15977, suffix heap size=142264,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=349 2023-04-17 07:59:55,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15974, totalHeapSize=36536, min timestamp=1681718395795, max timestamp=1681718395807Num uniques 1099; 2023-04-17 07:59:55,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=15974, suffix heap size=142264,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=349 2023-04-17 07:59:55,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1681718395795, max timestamp=1681718395807Num uniques 1009; 2023-04-17 07:59:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15984, suffix heap size=132904,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=349 2023-04-17 07:59:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:55,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@685e68bc[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:55,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:55,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:55,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:55,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:55,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:55,832 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,832 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:55,832 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,832 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:55,840 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:55,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:55,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:55,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:55,840 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:55,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:55,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,849 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/a88b5acb51964bb2be69c5fb12401de0 2023-04-17 07:59:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:55,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718395822, max timestamp=1681718395847Num uniques 1208; 2023-04-17 07:59:55,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=11810, suffix heap size=153600,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=258 2023-04-17 07:59:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:55,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681718395831, max timestamp=1681718395847Num uniques 1208; 2023-04-17 07:59:55,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-17 07:59:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681718395831, max timestamp=1681718395852Num uniques 1208; 2023-04-17 07:59:55,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=7230, suffix heap size=153600,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=158 2023-04-17 07:59:55,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:55,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:55,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:55,867 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:55,867 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:55,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:55,866 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:55,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:55,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1681718395867, max timestamp=1681718395895Num uniques 1067; 2023-04-17 07:59:55,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:55,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-17 07:59:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:58376 deadline: 1681718455892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14505, totalHeapSize=33208, min timestamp=1681718395867, max timestamp=1681718395895Num uniques 1017; 2023-04-17 07:59:55,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46565, new segment data size=14505, suffix heap size=133736,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=317 2023-04-17 07:59:55,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14505, totalHeapSize=33208, min timestamp=1681718395867, max timestamp=1681718395895Num uniques 1117; 2023-04-17 07:59:55,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=51145, new segment data size=14505, suffix heap size=144136,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=317 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718395919, max timestamp=1681718395924Num uniques 1176; 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=6870, suffix heap size=150272,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=150 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681718395911, max timestamp=1681718395924Num uniques 1176; 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681718395911, max timestamp=1681718395924Num uniques 1176; 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=12636, suffix heap size=150272,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=276 2023-04-17 07:59:55,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=12636, suffix heap size=150272,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=276 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681718395938, max timestamp=1681718395951Num uniques 1135; 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681718395938, max timestamp=1681718395951Num uniques 1009; 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51970, new segment data size=8460, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681718395938, max timestamp=1681718395951Num uniques 1135; 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=51970, new segment data size=8460, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-17 07:59:55,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=8460, suffix heap size=132904,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=185 2023-04-17 07:59:55,967 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:55,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:55,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:55,973 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:55,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:55,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:55,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,982 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:55,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:55,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13454, totalHeapSize=30816, min timestamp=1681718395971, max timestamp=1681718395981Num uniques 1044; 2023-04-17 07:59:55,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47804, new segment data size=13454, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-17 07:59:55,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13459, totalHeapSize=30816, min timestamp=1681718395971, max timestamp=1681718395981Num uniques 1044; 2023-04-17 07:59:55,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47809, new segment data size=13459, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-17 07:59:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13454, totalHeapSize=30816, min timestamp=1681718395971, max timestamp=1681718395981Num uniques 1044; 2023-04-17 07:59:55,985 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:55,985 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:55,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47804, new segment data size=13454, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-17 07:59:55,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:55,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:55,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:55,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:55,986 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:55,986 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:55,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1681718395990, max timestamp=1681718396008Num uniques 1153; 2023-04-17 07:59:56,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1681718395990, max timestamp=1681718396008Num uniques 1153; 2023-04-17 07:59:56,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=16165, suffix heap size=147880,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=353 2023-04-17 07:59:56,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=16165, suffix heap size=147880,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=353 2023-04-17 07:59:56,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681718395997, max timestamp=1681718396010Num uniques 1153; 2023-04-17 07:59:56,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=7005, suffix heap size=147880,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=153 2023-04-17 07:59:56,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,016 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,016 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:56,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d9c8af[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:56,018 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:56,018 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,024 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:56,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:56,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:56,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:56,032 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:56,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:56,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:56,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:56,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:56,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:56,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:56,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681718396020, max timestamp=1681718396036Num uniques 1212; 2023-04-17 07:59:56,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681718396020, max timestamp=1681718396036Num uniques 1212; 2023-04-17 07:59:56,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=55502, new segment data size=14282, suffix heap size=154016,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1212, new segment cells count=312 2023-04-17 07:59:56,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=55502, new segment data size=14282, suffix heap size=154016,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1212, new segment cells count=312 2023-04-17 07:59:56,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681718396020, max timestamp=1681718396036Num uniques 1012; 2023-04-17 07:59:56,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=14282, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-04-17 07:59:56,049 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,049 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,049 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,052 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,066 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,066 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,066 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,066 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:56,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:56,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:56,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681718396047, max timestamp=1681718396065Num uniques 1171; 2023-04-17 07:59:56,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=16986, suffix heap size=149752,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=371 2023-04-17 07:59:56,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681718396047, max timestamp=1681718396065Num uniques 1171; 2023-04-17 07:59:56,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681718396047, max timestamp=1681718396065Num uniques 1171; 2023-04-17 07:59:56,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=16986, suffix heap size=149752,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=371 2023-04-17 07:59:56,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=16986, suffix heap size=149752,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=371 2023-04-17 07:59:56,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681718396106, max timestamp=1681718396114Num uniques 1230; 2023-04-17 07:59:56,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=10530, suffix heap size=155888,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=230 2023-04-17 07:59:56,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681718396106, max timestamp=1681718396114Num uniques 1230; 2023-04-17 07:59:56,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=10530, suffix heap size=155888,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=230 2023-04-17 07:59:56,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681718396106, max timestamp=1681718396114Num uniques 1230; 2023-04-17 07:59:56,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=10530, suffix heap size=155888,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=230 2023-04-17 07:59:56,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681718396135, max timestamp=1681718396141Num uniques 1089; 2023-04-17 07:59:56,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-17 07:59:56,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681718396135, max timestamp=1681718396141Num uniques 1089; 2023-04-17 07:59:56,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-17 07:59:56,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681718396135, max timestamp=1681718396141Num uniques 1089; 2023-04-17 07:59:56,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-17 07:59:56,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681718396158, max timestamp=1681718396164Num uniques 1098; 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681718396158, max timestamp=1681718396164Num uniques 1098; 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681718396158, max timestamp=1681718396164Num uniques 1098; 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-17 07:59:56,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-17 07:59:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:56,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:56,173 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:56,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:56,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:56,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:56,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:56,184 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:56,184 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,184 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681718396179, max timestamp=1681718396190Num uniques 1107; 2023-04-17 07:59:56,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-17 07:59:56,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681718396179, max timestamp=1681718396190Num uniques 1107; 2023-04-17 07:59:56,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-17 07:59:56,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681718396179, max timestamp=1681718396190Num uniques 1107; 2023-04-17 07:59:56,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-17 07:59:56,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,196 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,196 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,207 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:56,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@657dcc19[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:56,209 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:56,209 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681718396199, max timestamp=1681718396210Num uniques 1066; 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681718396199, max timestamp=1681718396210Num uniques 1066; 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681718396199, max timestamp=1681718396210Num uniques 1066; 2023-04-17 07:59:56,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-17 07:59:56,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:56,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:56,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:56,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:56,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:56,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:56,228 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:56,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:56,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:56,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:56,228 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:56,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681718396221, max timestamp=1681718396228Num uniques 1125; 2023-04-17 07:59:56,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=17170, suffix heap size=144968,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=375 2023-04-17 07:59:56,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681718396221, max timestamp=1681718396228Num uniques 1125; 2023-04-17 07:59:56,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681718396221, max timestamp=1681718396228Num uniques 1125; 2023-04-17 07:59:56,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=8010, suffix heap size=144968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=175 2023-04-17 07:59:56,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=17170, suffix heap size=144968,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=375 2023-04-17 07:59:56,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1681718396247, max timestamp=1681718396254Num uniques 1234; 2023-04-17 07:59:56,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=56504, new segment data size=8414, suffix heap size=156304,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=184 2023-04-17 07:59:56,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1681718396247, max timestamp=1681718396254Num uniques 1234; 2023-04-17 07:59:56,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=56504, new segment data size=8414, suffix heap size=156304,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=184 2023-04-17 07:59:56,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1681718396247, max timestamp=1681718396254Num uniques 1034; 2023-04-17 07:59:56,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47344, new segment data size=8414, suffix heap size=135504,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=184 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:56,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,266 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,267 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:56,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/f99bc82c43bf4344813248257e6dbad6 2023-04-17 07:59:56,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/3d01738eabfd4d8a9f24537db00ae79b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b 2023-04-17 07:59:56,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b, entries=200, sequenceid=211, filesize=14.1 K 2023-04-17 07:59:56,304 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/a88b5acb51964bb2be69c5fb12401de0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718396267, max timestamp=1681718396278Num uniques 1043; 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=7688, suffix heap size=136440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=168 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8820, totalHeapSize=20312, min timestamp=1681718396267, max timestamp=1681718396278Num uniques 1043; 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8822, totalHeapSize=20312, min timestamp=1681718396267, max timestamp=1681718396278Num uniques 1043; 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47750, new segment data size=8820, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-17 07:59:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47752, new segment data size=8822, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-17 07:59:56,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0, entries=150, sequenceid=211, filesize=11.8 K 2023-04-17 07:59:56,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/f99bc82c43bf4344813248257e6dbad6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6 2023-04-17 07:59:56,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6, entries=150, sequenceid=211, filesize=11.8 K 2023-04-17 07:59:56,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~62.81 KB/64320, currentSize=59.27 KB/60690 for 2a1678219a4c91eb2f28c4b713537f8a in 1016ms, sequenceid=211, compaction requested=false 2023-04-17 07:59:56,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:56,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=65.98 KB heapSize=180.04 KB 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 07:59:56,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:56,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718456362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718456363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718456364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:56,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718456368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718456368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:56,379 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:56,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:56,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:56,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/3177b3e8d1fc43329a8ca83772148510 2023-04-17 07:59:56,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,392 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,392 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:56,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:56,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:56,394 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:56,394 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,394 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,396 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,401 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,401 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,401 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,401 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,401 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:56,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6827d664[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:56,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:56,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,424 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,424 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:56,424 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:56,424 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:56,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:56,433 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:56,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,433 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:56,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:56,438 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:56,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:56,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:56,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:56,439 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:56,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,463 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:56,463 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:56,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718456470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718456470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718456471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718456476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718456488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,563 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:56,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:56,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:56,584 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:56,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:56,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:56,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,599 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:56,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:56,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:56,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:56,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,608 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,608 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,616 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:56,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73274854[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:56,618 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:56,618 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:56,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:56,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:56,630 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,630 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:56,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:56,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:56,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:56,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:56,645 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:56,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:56,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:56,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:56,645 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:56,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,655 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,666 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:56,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:56,666 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:56,666 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718456675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718456676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718456676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718456684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718456694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,766 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:56,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:56,771 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:56,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:56,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:56,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:56,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:56,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:56,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:56,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,788 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,788 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,788 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,788 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,799 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,799 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:56,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e166a2f[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:56,801 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:56,801 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:56,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:56,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:56,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:56,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:56,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:56,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:56,818 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:56,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:56,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:56,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:56,818 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:56,823 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,824 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,824 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:56,824 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/0b4ccca125414363809988c5ce15f99d 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:56,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:56,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:56,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:56,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,868 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:56,969 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:56,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:56,976 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:56,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:56,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:56,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718456980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718456983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:56,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:56,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:56,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718456989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:56,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:56,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:56,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:56,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:56,992 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,992 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:56,992 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:56,992 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:56,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:56,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:56,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718456992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718456999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,004 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:57,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@558068f0[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:57,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:57,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,016 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:57,016 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:57,016 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:57,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:57,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:57,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:57,024 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:57,024 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:57,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:57,027 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:57,027 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:57,027 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:57,027 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:57,027 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:57,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:57,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:57,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:57,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:57,058 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:57,058 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:57,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:57,158 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:57,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:57,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:57,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:57,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:57,163 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:57,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:57,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:57,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:57,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:57,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:57,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:57,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:57,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:57,176 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:57,176 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:57,176 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:57,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:57,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:57,178 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,178 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:57,179 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:57,179 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:57,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:57,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:57,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:57,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,188 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:57,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34200d2[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:57,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:57,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:57,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:57,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:57,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:57,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:57,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:57,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:57,207 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:57,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:57,213 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:57,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:57,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:57,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:57,213 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:57,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:57,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:57,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:57,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:57,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:57,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:57,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:57,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:57,238 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:57,238 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:57,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:57,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:57,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=21.25 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5e2fffa24e7d44e798d364a8f9308ba3 2023-04-17 07:59:57,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/3177b3e8d1fc43329a8ca83772148510 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510 2023-04-17 07:59:57,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510, entries=250, sequenceid=686, filesize=16.7 K 2023-04-17 07:59:57,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/0b4ccca125414363809988c5ce15f99d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d 2023-04-17 07:59:57,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d, entries=150, sequenceid=686, filesize=11.9 K 2023-04-17 07:59:57,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5e2fffa24e7d44e798d364a8f9308ba3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3 2023-04-17 07:59:57,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3, entries=150, sequenceid=686, filesize=11.9 K 2023-04-17 07:59:57,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~65.98 KB/67560, heapSize ~179.99 KB/184312, currentSize=114.05 KB/116790 for 2a1678219a4c91eb2f28c4b713537f8a in 973ms, sequenceid=686, compaction requested=true 2023-04-17 07:59:57,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:57,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:57,311 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:57,311 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:57,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:57,313 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:57,313 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:57,313 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 07:59:57,313 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 07:59:57,313 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,313 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,313 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ec7f9684da614d219e5153cd1e4d49e4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=35.5 K 2023-04-17 07:59:57,313 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0f0f12938d724529b09edbaec2446f65, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=42.6 K 2023-04-17 07:59:57,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ec7f9684da614d219e5153cd1e4d49e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:57,314 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0f0f12938d724529b09edbaec2446f65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:57,318 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3d01738eabfd4d8a9f24537db00ae79b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681718394858 2023-04-17 07:59:57,318 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a88b5acb51964bb2be69c5fb12401de0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681718394871 2023-04-17 07:59:57,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3177b3e8d1fc43329a8ca83772148510, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:57,320 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 0b4ccca125414363809988c5ce15f99d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:57,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:57,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:57,346 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#30 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:57,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:57,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:57,352 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:57,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:57,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:57,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:57,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,355 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:57,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,362 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:57,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:57,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:57,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:57,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:57,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:57,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:57,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:57,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:57,392 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/37fb56c00aff4c61af921c9727547a33 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/37fb56c00aff4c61af921c9727547a33 2023-04-17 07:59:57,409 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:57,409 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:57,409 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,410 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:57,411 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:57,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:57,415 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into 37fb56c00aff4c61af921c9727547a33(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:57,416 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:57,416 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718397311; duration=0sec 2023-04-17 07:59:57,416 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:57,417 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:57,418 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:57,419 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 07:59:57,419 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:57,419 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c6cff3b039b64056b20612a1f4597f0b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=35.5 K 2023-04-17 07:59:57,419 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting c6cff3b039b64056b20612a1f4597f0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681718394276 2023-04-17 07:59:57,420 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f99bc82c43bf4344813248257e6dbad6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681718394871 2023-04-17 07:59:57,420 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/52ad8118c08043c6844ccd5d6484e3d5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/52ad8118c08043c6844ccd5d6484e3d5 2023-04-17 07:59:57,422 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5e2fffa24e7d44e798d364a8f9308ba3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:57,429 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 52ad8118c08043c6844ccd5d6484e3d5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:57,430 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:57,430 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718397311; duration=0sec 2023-04-17 07:59:57,431 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:57,436 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:57,452 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/11dcecf9eba24bbdbb7343af9d4bdeb1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/11dcecf9eba24bbdbb7343af9d4bdeb1 2023-04-17 07:59:57,460 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 11dcecf9eba24bbdbb7343af9d4bdeb1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:57,460 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:57,460 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718397312; duration=0sec 2023-04-17 07:59:57,460 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:57,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:57,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:57,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718457495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718457497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718457499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718457500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718457504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718457598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718457602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718457802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:57,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718457805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:57,828 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/c127700a53e8453fad86d48e0d96d33e 2023-04-17 07:59:57,848 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/62309854370646f1a86bc7e404c3de30 2023-04-17 07:59:58,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718458106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718458106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,264 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/41ac72008d6a48d68df611ade23e50ee 2023-04-17 07:59:58,272 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/c127700a53e8453fad86d48e0d96d33e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e 2023-04-17 07:59:58,278 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e, entries=150, sequenceid=706, filesize=11.9 K 2023-04-17 07:59:58,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/62309854370646f1a86bc7e404c3de30 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30 2023-04-17 07:59:58,287 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30, entries=150, sequenceid=706, filesize=11.9 K 2023-04-17 07:59:58,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/41ac72008d6a48d68df611ade23e50ee as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee 2023-04-17 07:59:58,295 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee, entries=150, sequenceid=706, filesize=11.9 K 2023-04-17 07:59:58,301 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2a1678219a4c91eb2f28c4b713537f8a in 891ms, sequenceid=706, compaction requested=false 2023-04-17 07:59:58,301 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:58,301 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,301 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:58,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:58,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:58,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:58,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:58,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:58,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6e8d06[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:58,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,357 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:58,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:58,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:58,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:58,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:58,365 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:58,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:58,367 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:58,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:58,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:58,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:58,367 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:58,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,375 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:58,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:58,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,383 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,383 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,482 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:58,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:58,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:58,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:58,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:58,487 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:58,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:58,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:58,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:58,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:58,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:58,518 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,518 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,519 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,519 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:58,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:58,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:58,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:58,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:58,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:58,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:58,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:58,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2744d176[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:58,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:58,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,540 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:58,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:58,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:58,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:58,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:58,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,549 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:58,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:58,551 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:58,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:58,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:58,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:58,552 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:58,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718458551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718458551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718458555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,574 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,574 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718458610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718458610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718458657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718458657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718458663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:58,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:58,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:58,678 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:58,678 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,678 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:58,678 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:58,678 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,678 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,688 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:58,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:58,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:58,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:58,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:58,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:58,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:58,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:58,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d5fa86[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:58,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:58,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:58,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:58,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:58,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:58,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:58,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:58,717 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:58,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:58,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:58,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:58,717 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:58,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,731 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:58,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:58,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,741 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:58,741 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,741 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,841 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:58,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:58,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:58,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:58,845 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:58,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:58,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:58,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,854 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:58,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:58,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:58,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:58,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:58,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:58,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:58,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:58,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:58,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:58,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718458860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718458861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,866 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@628220fe[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:58,867 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:58,867 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718458865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:58,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:58,882 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:58,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:58,884 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:58,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:58,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:58,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:58,884 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:58,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:58,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:58,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:58,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:58,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:58,899 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:58,899 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:58,899 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:58,900 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:58,900 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:58,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d91b3874ab964c31b03fee07f2382ecb 2023-04-17 07:59:58,999 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:59,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:59,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:59,005 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:59,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:59,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:59,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:59,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:59,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:59,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:59,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:59,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:59,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:59,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:59,042 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,042 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,043 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,043 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cba9715[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:59,044 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:59,044 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:59,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:59,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:59,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:59,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:59,068 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:59,068 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:59,068 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:59,068 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:59,068 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:59,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,074 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,075 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:59,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:59,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,083 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,083 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,083 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,083 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:59,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,083 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:59,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718459163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718459165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718459171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:59,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:59,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:59,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:59,187 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:59,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:59,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:59,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:59,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:59,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:59,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:59,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:59,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:59,199 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,199 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:59,199 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:59,199 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:59,199 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:59,207 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:59,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d743295[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:59,210 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:59,210 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:59,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:59,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:59,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:59,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:59,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:59,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:59,235 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:59,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:59,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:59,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:59,235 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:59,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:59,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:59,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:59,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:59,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:59,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/09f2b3da60404582907df3142027d080 2023-04-17 07:59:59,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:59,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:59,364 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:59,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/4872471ed3234d6aafc7e8ce98d06d8e 2023-04-17 07:59:59,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:59,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:59,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d91b3874ab964c31b03fee07f2382ecb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb 2023-04-17 07:59:59,374 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:59,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:59,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb, entries=200, sequenceid=722, filesize=14.3 K 2023-04-17 07:59:59,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/09f2b3da60404582907df3142027d080 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080 2023-04-17 07:59:59,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:59,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:59,392 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:59,392 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,392 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:59,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080, entries=150, sequenceid=722, filesize=11.9 K 2023-04-17 07:59:59,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/4872471ed3234d6aafc7e8ce98d06d8e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:59,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:59,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:59,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:59,400 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 07:59:59,400 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 07:59:59,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e, entries=150, sequenceid=722, filesize=11.9 K 2023-04-17 07:59:59,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2a1678219a4c91eb2f28c4b713537f8a in 894ms, sequenceid=722, compaction requested=true 2023-04-17 07:59:59,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:59,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:59,405 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:59,405 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:59,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:59,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:59,407 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:59,407 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,407 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 07:59:59,407 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,407 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/52ad8118c08043c6844ccd5d6484e3d5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=38.4 K 2023-04-17 07:59:59,407 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:59,407 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 07:59:59,407 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,408 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/37fb56c00aff4c61af921c9727547a33, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.0 K 2023-04-17 07:59:59,408 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,408 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 37fb56c00aff4c61af921c9727547a33, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:59,408 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 52ad8118c08043c6844ccd5d6484e3d5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,416 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 62309854370646f1a86bc7e404c3de30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1681718396362 2023-04-17 07:59:59,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 07:59:59,417 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting c127700a53e8453fad86d48e0d96d33e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1681718396362 2023-04-17 07:59:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,417 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 09f2b3da60404582907df3142027d080, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397494 2023-04-17 07:59:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,422 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d91b3874ab964c31b03fee07f2382ecb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397493 2023-04-17 07:59:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 07:59:59,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3abf277[Count = 0] remaining members to acquire global barrier 2023-04-17 07:59:59,427 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 07:59:59,427 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 07:59:59,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 07:59:59,437 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 07:59:59,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 07:59:59,441 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:59,447 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:59,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,450 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 07:59:59,450 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,450 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 07:59:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 07:59:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 07:59:59,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 07:59:59,452 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 07:59:59,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 07:59:59,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 07:59:59,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 07:59:59,452 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 07:59:59,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,468 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 07:59:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 07:59:59,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 07:59:59,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 07:59:59,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 07:59:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 07:59:59,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 07:59:59,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,475 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,475 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,475 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 07:59:59,475 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 07:59:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 07:59:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 07:59:59,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 07:59:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 07:59:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 07:59:59,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 07:59:59,494 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/7dbd4c0a887446f7a3b76eb8b99d3863 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/7dbd4c0a887446f7a3b76eb8b99d3863 2023-04-17 07:59:59,497 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/afedf69be11c47daa2cd3b957c97e544 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/afedf69be11c47daa2cd3b957c97e544 2023-04-17 07:59:59,504 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 7dbd4c0a887446f7a3b76eb8b99d3863(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:59,505 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:59,505 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718399405; duration=0sec 2023-04-17 07:59:59,505 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 07:59:59,505 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 07:59:59,505 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into afedf69be11c47daa2cd3b957c97e544(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:59,505 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:59,506 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718399405; duration=0sec 2023-04-17 07:59:59,506 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:59,507 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 07:59:59,507 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 07:59:59,507 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,507 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/11dcecf9eba24bbdbb7343af9d4bdeb1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.0 K 2023-04-17 07:59:59,508 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 11dcecf9eba24bbdbb7343af9d4bdeb1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681718396274 2023-04-17 07:59:59,508 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 41ac72008d6a48d68df611ade23e50ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1681718396362 2023-04-17 07:59:59,509 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4872471ed3234d6aafc7e8ce98d06d8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397494 2023-04-17 07:59:59,519 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#41 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 07:59:59,538 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/26e74304e023480e99e72c4f61a66a5e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/26e74304e023480e99e72c4f61a66a5e 2023-04-17 07:59:59,545 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 26e74304e023480e99e72c4f61a66a5e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 07:59:59,545 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 07:59:59,545 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718399405; duration=0sec 2023-04-17 07:59:59,545 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 07:59:59,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 07:59:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 07:59:59,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 07:59:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 07:59:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 07:59:59,579 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 07:59:59,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 07:59:59,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 07:59:59,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 07:59:59,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 07:59:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 07:59:59,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 07:59:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 07:59:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 07:59:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 07:59:59,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 07:59:59,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 07:59:59,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 07:59:59,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 07:59:59,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 07:59:59,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 07:59:59,594 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 07:59:59,594 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 07:59:59,594 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 07:59:59,594 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:59,595 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 07:59:59,595 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 07:59:59,609 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/13cfbe2ff23f49ddb0acbf20099efe42 2023-04-17 07:59:59,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 07:59:59,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 07:59:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718459633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718459633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718459668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718459669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718459675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718459735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718459735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718459938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 07:59:59,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 07:59:59,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718459940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,028 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2a6fe5182c5c46b79ab01fee0095b9c4 2023-04-17 08:00:00,042 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5764439bc0564fdab56246b9765dbc0e 2023-04-17 08:00:00,049 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/13cfbe2ff23f49ddb0acbf20099efe42 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42 2023-04-17 08:00:00,055 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42, entries=150, sequenceid=744, filesize=11.9 K 2023-04-17 08:00:00,064 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2a6fe5182c5c46b79ab01fee0095b9c4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4 2023-04-17 08:00:00,070 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4, entries=150, sequenceid=744, filesize=11.9 K 2023-04-17 08:00:00,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5764439bc0564fdab56246b9765dbc0e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e 2023-04-17 08:00:00,079 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e, entries=150, sequenceid=744, filesize=11.9 K 2023-04-17 08:00:00,080 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2a1678219a4c91eb2f28c4b713537f8a in 485ms, sequenceid=744, compaction requested=false 2023-04-17 08:00:00,080 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:00,080 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:00,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:00,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:00,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:00,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705db72c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:00,120 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:00,120 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:00,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:00,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:00,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:00,162 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:00,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:00,164 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:00,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:00,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:00,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:00,164 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:00,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,174 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:00,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,175 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:00,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:00,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:00,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:00,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718460266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718460268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/5e41ecf8db374dc994d443fdddbd74c7 2023-04-17 08:00:00,292 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:00,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:00,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:00,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:00,296 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:00,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:00,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:00,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:00,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:00,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:00,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:00,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:00,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:00,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:00,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:00,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:00,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:00,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:00,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/899eb9adb4ff4574867026c11e9eb59c 2023-04-17 08:00:00,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:00,331 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:00,331 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:00,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:00,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:00,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:00,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,341 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,341 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:00,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d7940d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:00,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:00,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:00,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:00,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:00,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:00,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/ef6064f0d09e4473b4125eb8f508c90d 2023-04-17 08:00:00,357 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:00,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:00,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/5e41ecf8db374dc994d443fdddbd74c7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7 2023-04-17 08:00:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:00,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:00,361 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:00,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:00,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:00,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:00,362 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:00,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7, entries=200, sequenceid=759, filesize=14.3 K 2023-04-17 08:00:00,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/899eb9adb4ff4574867026c11e9eb59c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c 2023-04-17 08:00:00,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718460370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:00,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:00,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:00,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718460370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,378 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c, entries=150, sequenceid=759, filesize=11.9 K 2023-04-17 08:00:00,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/ef6064f0d09e4473b4125eb8f508c90d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d 2023-04-17 08:00:00,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:00,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:00,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:00,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:00,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:00,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:00,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:00,391 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,391 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:00,391 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,391 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,391 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:00,391 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d, entries=150, sequenceid=759, filesize=11.9 K 2023-04-17 08:00:00,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2a1678219a4c91eb2f28c4b713537f8a in 153ms, sequenceid=759, compaction requested=true 2023-04-17 08:00:00,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:00,397 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:00,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:00,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:00,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 08:00:00,397 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:00,398 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:00,399 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:00,399 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:00,399 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,399 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:00,399 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,399 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/7dbd4c0a887446f7a3b76eb8b99d3863, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=38.5 K 2023-04-17 08:00:00,399 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/afedf69be11c47daa2cd3b957c97e544, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.1 K 2023-04-17 08:00:00,399 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7dbd4c0a887446f7a3b76eb8b99d3863, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397494 2023-04-17 08:00:00,399 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting afedf69be11c47daa2cd3b957c97e544, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397494 2023-04-17 08:00:00,400 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 13cfbe2ff23f49ddb0acbf20099efe42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1681718398523 2023-04-17 08:00:00,400 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2a6fe5182c5c46b79ab01fee0095b9c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1681718398523 2023-04-17 08:00:00,400 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5e41ecf8db374dc994d443fdddbd74c7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:00,400 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 899eb9adb4ff4574867026c11e9eb59c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:00,415 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:00,415 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:00,491 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:00,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:00,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:00,495 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:00,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:00,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:00,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:00,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:00,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:00,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:00,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:00,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:00,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:00,506 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:00,506 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:00,506 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:00,512 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,512 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:00,512 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:00,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:00,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:00,530 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/5ef05d4c2282468da808a0d83c96da60 2023-04-17 08:00:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:00,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:00,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718460588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718460589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718460675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718460678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718460683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718460690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718460691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,835 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/d0d46495871f4d7d935630a81bbb3a9b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d0d46495871f4d7d935630a81bbb3a9b 2023-04-17 08:00:00,837 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/a06944de1e1846aaabc9e090064e7080 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a06944de1e1846aaabc9e090064e7080 2023-04-17 08:00:00,843 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into d0d46495871f4d7d935630a81bbb3a9b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:00,843 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into a06944de1e1846aaabc9e090064e7080(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:00,843 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:00,843 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:00,843 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718400397; duration=0sec 2023-04-17 08:00:00,843 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718400397; duration=0sec 2023-04-17 08:00:00,844 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:00,844 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:00,844 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:00,845 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:00,845 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:00,845 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:00,845 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/26e74304e023480e99e72c4f61a66a5e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.1 K 2023-04-17 08:00:00,849 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 26e74304e023480e99e72c4f61a66a5e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681718397494 2023-04-17 08:00:00,850 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5764439bc0564fdab56246b9765dbc0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1681718398523 2023-04-17 08:00:00,850 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ef6064f0d09e4473b4125eb8f508c90d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:00,857 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:00,884 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/2f0ce7dfff814c40921d06c73aed3404 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2f0ce7dfff814c40921d06c73aed3404 2023-04-17 08:00:00,895 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 2f0ce7dfff814c40921d06c73aed3404(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:00,895 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:00,895 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718400397; duration=0sec 2023-04-17 08:00:00,895 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:00,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718460893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718460894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:00,946 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/ae8d22914be84e14a51403ccdc698e28 2023-04-17 08:00:00,998 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/8ebc91a5197948eda10db4cd2f14a7bc 2023-04-17 08:00:01,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/5ef05d4c2282468da808a0d83c96da60 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60 2023-04-17 08:00:01,018 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60, entries=150, sequenceid=776, filesize=11.9 K 2023-04-17 08:00:01,020 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/ae8d22914be84e14a51403ccdc698e28 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28 2023-04-17 08:00:01,027 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28, entries=150, sequenceid=776, filesize=11.9 K 2023-04-17 08:00:01,028 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/8ebc91a5197948eda10db4cd2f14a7bc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc 2023-04-17 08:00:01,036 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc, entries=150, sequenceid=776, filesize=11.9 K 2023-04-17 08:00:01,037 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2a1678219a4c91eb2f28c4b713537f8a in 525ms, sequenceid=776, compaction requested=false 2023-04-17 08:00:01,037 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:01,037 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cbe118[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:01,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:01,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:01,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:01,092 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:01,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:01,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:01,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:01,092 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:01,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:01,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:01,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,116 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:01,116 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:01,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:01,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:01,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,216 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:01,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:01,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:01,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:01,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:01,220 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:01,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:01,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:01,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718461223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718461223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/99d5cf44663e4fae848df4093fcea496 2023-04-17 08:00:01,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:01,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:01,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:01,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:01,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:01,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:01,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:01,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:01,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:01,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,240 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,241 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,241 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2cddeaf6f70e48428b91296064adee8e 2023-04-17 08:00:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582f83bc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,249 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,249 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,249 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:01,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/a41fd54e8cd740119136f646bbfcd342 2023-04-17 08:00:01,257 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:01,257 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,257 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:01,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/99d5cf44663e4fae848df4093fcea496 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496 2023-04-17 08:00:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:01,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:01,260 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:01,260 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:01,260 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:01,260 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:01,260 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:01,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496, entries=200, sequenceid=795, filesize=14.3 K 2023-04-17 08:00:01,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/2cddeaf6f70e48428b91296064adee8e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e 2023-04-17 08:00:01,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e, entries=150, sequenceid=795, filesize=11.9 K 2023-04-17 08:00:01,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/a41fd54e8cd740119136f646bbfcd342 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342 2023-04-17 08:00:01,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342, entries=150, sequenceid=795, filesize=11.9 K 2023-04-17 08:00:01,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 2a1678219a4c91eb2f28c4b713537f8a in 82ms, sequenceid=795, compaction requested=true 2023-04-17 08:00:01,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:01,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:01,282 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:01,282 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:01,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:01,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:01,286 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:01,286 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:01,286 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:01,286 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:01,286 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,286 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,286 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a06944de1e1846aaabc9e090064e7080, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=38.6 K 2023-04-17 08:00:01,286 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d0d46495871f4d7d935630a81bbb3a9b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.2 K 2023-04-17 08:00:01,287 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a06944de1e1846aaabc9e090064e7080, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:01,287 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d0d46495871f4d7d935630a81bbb3a9b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:01,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,288 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5ef05d4c2282468da808a0d83c96da60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681718400264 2023-04-17 08:00:01,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,296 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 99d5cf44663e4fae848df4093fcea496, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:01,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ae8d22914be84e14a51403ccdc698e28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681718400264 2023-04-17 08:00:01,297 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2cddeaf6f70e48428b91296064adee8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:01,303 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:01,303 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:01,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:01,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:01,305 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:01,305 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:01,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,304 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,323 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/d7d391b8643f48c4b2920c2c05337057 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7d391b8643f48c4b2920c2c05337057 2023-04-17 08:00:01,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:01,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:00:01,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:01,332 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into d7d391b8643f48c4b2920c2c05337057(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:01,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,332 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:01,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:01,332 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718401282; duration=0sec 2023-04-17 08:00:01,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:01,333 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:01,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:01,333 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:01,335 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:01,335 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:01,335 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,335 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2f0ce7dfff814c40921d06c73aed3404, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.2 K 2023-04-17 08:00:01,336 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2f0ce7dfff814c40921d06c73aed3404, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1681718399629 2023-04-17 08:00:01,338 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8ebc91a5197948eda10db4cd2f14a7bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681718400264 2023-04-17 08:00:01,338 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a41fd54e8cd740119136f646bbfcd342, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:01,362 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:01,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718461363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718461364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,405 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:01,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:01,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:01,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:01,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:01,409 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:01,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:01,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:01,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:01,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:01,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:01,416 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:01,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,417 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:01,426 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:01,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:01,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:01,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,427 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d6bb42[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,449 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,449 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:01,449 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,457 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,457 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,457 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:01,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:01,459 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:01,459 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:01,459 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:01,459 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:01,459 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:01,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718461464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,466 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,468 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718461470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:01,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:01,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:01,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:01,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:01,580 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:01,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:01,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:01,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:01,583 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:01,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:01,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:01,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:01,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:01,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:01,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:01,591 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:01,591 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,591 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:01,592 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,592 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:01,592 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:01,592 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,592 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1653809[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:01,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:01,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:01,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:01,619 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:01,619 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:01,619 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:01,619 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:01,619 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:01,649 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,650 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,650 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,652 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718461667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,671 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:01,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,671 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:01,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:01,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718461672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,723 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/4e4217fe19f24a95bb964718e71b9c59 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/4e4217fe19f24a95bb964718e71b9c59 2023-04-17 08:00:01,731 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 4e4217fe19f24a95bb964718e71b9c59(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:01,731 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:01,731 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718401282; duration=0sec 2023-04-17 08:00:01,731 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:01,771 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:01,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:01,774 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:01,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:01,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:01,775 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,775 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/90ca1e1d5c6440f19e42699f4eaeef54 2023-04-17 08:00:01,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,783 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,783 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,783 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:01,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:01,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:01,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:01,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:01,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:01,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:01,784 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,784 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:01,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:01,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,785 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,791 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/6f7a263d6c2540a79f5bfa0e993c9bfc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6f7a263d6c2540a79f5bfa0e993c9bfc 2023-04-17 08:00:01,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,797 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 6f7a263d6c2540a79f5bfa0e993c9bfc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:01,797 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:01,797 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718401282; duration=0sec 2023-04-17 08:00:01,797 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7dca1c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,800 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,800 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:01,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:01,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:01,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:01,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:01,818 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:01,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:01,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:01,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:01,818 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:01,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,829 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:01,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:01,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:01,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:01,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:01,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:01,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:01,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:01,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:01,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:01,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:01,960 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:01,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:01,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:01,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718461970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,973 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:01,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:01,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:01,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:01,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:01,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:01,975 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,975 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:01,975 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:01,975 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:01,975 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:01,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:01,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718461978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:01,985 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:01,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:01,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e6b908c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:01,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:01,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:01,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:01,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:01,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:01,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:01,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,007 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,013 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,013 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,024 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,024 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,028 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,041 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,042 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,042 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,043 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,043 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:02,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:02,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:02,045 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:02,145 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:02,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:02,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:02,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:02,147 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:02,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:02,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:02,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,157 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:02,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:02,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:02,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:02,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:02,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:02,159 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,159 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:02,159 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:02,159 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:02,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:02,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:02,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:02,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9738a8[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:02,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,174 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:02,174 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:02,174 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:02,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,184 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,184 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:02,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:02,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:02,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:02,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:02,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/365f6a46d4f94c0cadb67454557f1a78 2023-04-17 08:00:02,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:02,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:02,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:02,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:02,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:02,303 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:02,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:02,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:02,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:02,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:02,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:02,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:02,315 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,315 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:02,315 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:02,315 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:02,316 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:02,316 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,316 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:02,316 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:02,323 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,324 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:02,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:02,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@269afc03[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:02,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,332 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:02,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:02,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:02,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,342 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,342 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,376 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,376 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:02,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:02,376 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,377 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,377 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,377 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,377 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,377 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:02,377 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:02,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:02,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718462474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,477 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:02,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:02,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:02,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:02,483 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:02,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:02,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:02,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718462482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,484 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,484 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:02,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:02,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:02,498 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,498 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:02,498 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:02,498 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,498 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:02,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:02,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:02,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1203ffe6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:02,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:02,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,515 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:02,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:02,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:02,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,532 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,534 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,534 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,549 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:02,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:02,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:02,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:02,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/7db7f35b75ad475cae5606c72b2c2f33 2023-04-17 08:00:02,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/90ca1e1d5c6440f19e42699f4eaeef54 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54 2023-04-17 08:00:02,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54, entries=250, sequenceid=815, filesize=16.7 K 2023-04-17 08:00:02,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/365f6a46d4f94c0cadb67454557f1a78 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78 2023-04-17 08:00:02,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78, entries=150, sequenceid=815, filesize=11.9 K 2023-04-17 08:00:02,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/7db7f35b75ad475cae5606c72b2c2f33 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33 2023-04-17 08:00:02,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33, entries=150, sequenceid=815, filesize=11.9 K 2023-04-17 08:00:02,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2a1678219a4c91eb2f28c4b713537f8a in 1313ms, sequenceid=815, compaction requested=false 2023-04-17 08:00:02,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:02,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:02,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:02,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:02,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:02,677 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:02,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:02,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:02,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:02,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 08:00:02,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:02,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:02,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:02,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:02,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:02,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:02,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718462706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718462707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718462708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,711 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:02,753 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:02,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:02,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:02,754 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,754 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:02,755 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:02,755 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:02,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:02,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:02,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:02,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,771 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,771 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:02,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:02,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30cef2da[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:02,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:02,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:02,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:02,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,799 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,799 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,799 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,799 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,799 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,807 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,808 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,809 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718462809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:02,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718462810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,821 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:02,821 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:02,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:02,922 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:02,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:02,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:02,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:02,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:02,932 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:02,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:02,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:02,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,949 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:02,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:02,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:02,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:02,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:02,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:02,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:02,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:02,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:02,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:02,954 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,954 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:02,954 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:02,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:02,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:02,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:02,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:02,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:02,966 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,967 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,967 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,967 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:02,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7abce13[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:02,969 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:02,969 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,979 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:02,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:02,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:02,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:02,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:02,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:02,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:02,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:02,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:02,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:02,990 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:02,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:02,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:02,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:02,990 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:02,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:02,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,001 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,013 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718463013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718463013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/edf0ba5e5e204947b6795b8060bc6c12 2023-04-17 08:00:03,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/92c1d8f97416405896b7dd1821e50d6c 2023-04-17 08:00:03,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:03,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:03,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:03,116 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:03,116 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,116 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:03,116 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:03,120 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,120 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/2a365a53a8db493090c9399b6182c2e9 2023-04-17 08:00:03,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:03,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:03,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:03,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:03,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:03,131 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,131 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:03,131 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:03,131 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:03,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:03,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/edf0ba5e5e204947b6795b8060bc6c12 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12 2023-04-17 08:00:03,144 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12, entries=150, sequenceid=832, filesize=11.9 K 2023-04-17 08:00:03,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/92c1d8f97416405896b7dd1821e50d6c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c 2023-04-17 08:00:03,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c, entries=150, sequenceid=832, filesize=11.9 K 2023-04-17 08:00:03,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/2a365a53a8db493090c9399b6182c2e9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9 2023-04-17 08:00:03,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,156 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:03,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76204001[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:03,158 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:03,158 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:03,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:03,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:03,171 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,171 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:03,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9, entries=150, sequenceid=832, filesize=11.9 K 2023-04-17 08:00:03,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2a1678219a4c91eb2f28c4b713537f8a in 492ms, sequenceid=832, compaction requested=true 2023-04-17 08:00:03,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:03,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:03,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:03,173 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:03,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:03,173 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:03,177 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:03,177 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42086 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:03,177 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:03,177 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:03,177 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,177 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/4e4217fe19f24a95bb964718e71b9c59, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=41.1 K 2023-04-17 08:00:03,177 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,178 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7d391b8643f48c4b2920c2c05337057, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.3 K 2023-04-17 08:00:03,178 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4e4217fe19f24a95bb964718e71b9c59, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:03,178 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d7d391b8643f48c4b2920c2c05337057, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:03,178 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 90ca1e1d5c6440f19e42699f4eaeef54, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681718401221 2023-04-17 08:00:03,178 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 365f6a46d4f94c0cadb67454557f1a78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681718401222 2023-04-17 08:00:03,179 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting edf0ba5e5e204947b6795b8060bc6c12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:03,179 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 92c1d8f97416405896b7dd1821e50d6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:03,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:03,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,182 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:03,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:03,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:03,187 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:03,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:03,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:03,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:03,187 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:03,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,201 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,201 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,203 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,210 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#67 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:03,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,231 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/8555144290fe4d358ec811e277772d5f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8555144290fe4d358ec811e277772d5f 2023-04-17 08:00:03,241 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 8555144290fe4d358ec811e277772d5f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:03,241 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:03,242 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718403172; duration=0sec 2023-04-17 08:00:03,242 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:03,242 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:03,244 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:03,244 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:03,244 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,244 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6f7a263d6c2540a79f5bfa0e993c9bfc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=36.3 K 2023-04-17 08:00:03,244 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6f7a263d6c2540a79f5bfa0e993c9bfc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681718400581 2023-04-17 08:00:03,245 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7db7f35b75ad475cae5606c72b2c2f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681718401222 2023-04-17 08:00:03,245 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2a365a53a8db493090c9399b6182c2e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:03,257 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:03,301 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/72e7ccc3b962440c87aef65210556807 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/72e7ccc3b962440c87aef65210556807 2023-04-17 08:00:03,309 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into 72e7ccc3b962440c87aef65210556807(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:03,309 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:03,309 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718403173; duration=0sec 2023-04-17 08:00:03,309 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:03,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:03,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:03,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:00:03,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:03,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:03,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:03,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:03,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:03,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:03,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:03,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:03,339 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:03,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:03,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:03,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718463349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718463349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,352 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:03,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:03,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:03,360 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:03,360 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,360 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:03,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:03,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:03,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,376 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:03,376 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:03,376 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,376 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:03,376 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:03,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:03,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b517dec[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:03,385 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:03,385 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:03,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:03,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:03,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:03,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:03,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:03,407 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:03,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:03,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:03,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:03,407 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:03,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,418 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,419 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,430 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,430 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718463451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718463453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718463476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718463487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,530 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:03,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:03,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:03,534 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:03,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:03,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:03,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,540 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:03,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:03,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:03,542 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,542 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:03,542 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:03,542 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:03,542 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:03,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,549 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,549 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:03,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cce187e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:03,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:03,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:03,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:03,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:03,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:03,565 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,565 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,565 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,565 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:03,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:03,568 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:03,568 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:03,568 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:03,568 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:03,568 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:03,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,592 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,592 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,632 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/c5870de5c64e4989bedd53c9cddcd1bd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c5870de5c64e4989bedd53c9cddcd1bd 2023-04-17 08:00:03,638 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into c5870de5c64e4989bedd53c9cddcd1bd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:03,638 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:03,638 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718403173; duration=0sec 2023-04-17 08:00:03,638 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:03,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718463656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718463657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,692 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:03,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:03,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:03,696 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:03,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:03,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:03,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:03,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:03,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:03,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:03,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:03,716 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,716 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:03,716 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:03,716 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:03,716 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:03,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:03,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cc92a2[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:03,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:03,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,749 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:03,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:03,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:03,749 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,749 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:03,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:03,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:03,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:03,764 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:03,764 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:03,764 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:03,764 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:03,764 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:03,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b2768eff2db241c7af915d5db013f335 2023-04-17 08:00:03,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,787 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,788 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,788 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,888 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:03,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:03,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:03,891 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:03,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:03,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:03,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,898 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:03,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:03,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:03,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:03,900 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,900 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:03,900 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:03,900 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:03,900 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:03,901 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:03,901 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,901 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:03,901 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:03,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,907 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:03,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1198e652[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:03,909 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:03,909 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,915 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:03,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:03,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:03,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:03,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,924 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:03,924 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:03,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:03,924 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:03,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:03,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:03,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:03,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,934 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:03,946 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:03,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:03,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:03,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:03,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:03,946 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:03,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:03,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:03,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:03,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718463960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:03,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718463961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:04,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:04,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:04,049 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:04,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:04,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:04,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:04,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,057 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:04,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:04,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:04,058 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:04,058 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,058 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:04,058 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:04,058 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:04,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:04,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:04,059 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:04,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:04,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:04,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:04,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:04,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61565790[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:04,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:04,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:04,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:04,074 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:04,074 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:04,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:04,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:04,083 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:04,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:04,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:04,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:04,084 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:04,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:04,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:04,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,099 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,099 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:04,099 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:04,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:04,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/4058db1f34094152981d12568e847f4c 2023-04-17 08:00:04,199 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:04,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:04,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:04,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:04,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:04,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:04,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:04,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:04,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/eae08552615d481ea20c0de64ca0d3b9 2023-04-17 08:00:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:04,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:04,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:04,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:04,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:04,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:04,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:04,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:04,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:04,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:04,221 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:04,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:04,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:04,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:04,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b2768eff2db241c7af915d5db013f335 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335 2023-04-17 08:00:04,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335, entries=150, sequenceid=853, filesize=11.9 K 2023-04-17 08:00:04,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/4058db1f34094152981d12568e847f4c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c 2023-04-17 08:00:04,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:04,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c, entries=150, sequenceid=853, filesize=11.9 K 2023-04-17 08:00:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/eae08552615d481ea20c0de64ca0d3b9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9 2023-04-17 08:00:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:04,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d17ae2c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:04,234 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:04,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9, entries=150, sequenceid=853, filesize=11.9 K 2023-04-17 08:00:04,240 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:04,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:04,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:04,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2a1678219a4c91eb2f28c4b713537f8a in 922ms, sequenceid=853, compaction requested=false 2023-04-17 08:00:04,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:04,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:04,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:04,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,266 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:04,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:04,269 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:04,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:04,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:04,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:04,269 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:04,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,284 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:04,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:04,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:04,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,291 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,291 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,291 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,291 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,291 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,291 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:04,291 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:04,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:04,391 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:04,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:04,394 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:04,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:04,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:04,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:04,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:04,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:04,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:04,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:04,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:04,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:04,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:04,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:04,407 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:04,407 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:04,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:04,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:04,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718464484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718464484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718464586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718464587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718464790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:04,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718464792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,817 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d324f3fb51f0444d904d035823b3e672 2023-04-17 08:00:04,853 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/dc33886ab60147168a17cfdd040db55a 2023-04-17 08:00:04,887 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e248e479f3a54e1c9319fa1c0c1b1281 2023-04-17 08:00:04,894 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d324f3fb51f0444d904d035823b3e672 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672 2023-04-17 08:00:04,901 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672, entries=150, sequenceid=868, filesize=11.9 K 2023-04-17 08:00:04,902 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/dc33886ab60147168a17cfdd040db55a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a 2023-04-17 08:00:04,909 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a, entries=150, sequenceid=868, filesize=11.9 K 2023-04-17 08:00:04,912 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e248e479f3a54e1c9319fa1c0c1b1281 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281 2023-04-17 08:00:04,917 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281, entries=150, sequenceid=868, filesize=11.9 K 2023-04-17 08:00:04,918 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2a1678219a4c91eb2f28c4b713537f8a in 511ms, sequenceid=868, compaction requested=true 2023-04-17 08:00:04,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:04,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:04,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:04,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:04,919 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:04,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:04,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:04,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ccf60ae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:04,958 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:04,958 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,965 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:04,966 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:04,966 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:04,966 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:04,973 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:04,974 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,973 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,974 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:04,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:04,977 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:04,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:04,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:04,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:04,977 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:04,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:04,989 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:04,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:04,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:04,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:04,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:04,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:04,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:04,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:04,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:05,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:05,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718404469, max timestamp=1681718405095Num uniques 859; 2023-04-17 08:00:05,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:05,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718404469, max timestamp=1681718405095Num uniques 859; 2023-04-17 08:00:05,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:05,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718404469, max timestamp=1681718405095Num uniques 859; 2023-04-17 08:00:05,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:05,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:05,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:05,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:05,104 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:05,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:05,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:05,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:05,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:05,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:05,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:05,115 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,115 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:05,115 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:05,115 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:05,115 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:05,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,124 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,124 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,124 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:05,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a33b913[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:05,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:05,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,132 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,132 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:05,132 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:05,132 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:05,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:05,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:05,140 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:05,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:05,143 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:05,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:05,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:05,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:05,143 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681718405106, max timestamp=1681718405145Num uniques 859; 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681718405106, max timestamp=1681718405145Num uniques 859; 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681718405106, max timestamp=1681718405145Num uniques 859; 2023-04-17 08:00:05,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 08:00:05,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:05,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:05,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:05,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:05,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:05,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718465177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718405169, max timestamp=1681718405178Num uniques 1168; 2023-04-17 08:00:05,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-17 08:00:05,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718405169, max timestamp=1681718405178Num uniques 1168; 2023-04-17 08:00:05,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718405169, max timestamp=1681718405178Num uniques 1168; 2023-04-17 08:00:05,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-17 08:00:05,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718405217, max timestamp=1681718405249Num uniques 1027; 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718405217, max timestamp=1681718405249Num uniques 1027; 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718405217, max timestamp=1681718405249Num uniques 1027; 2023-04-17 08:00:05,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:00:05,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:05,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:05,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:05,272 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:05,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:05,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:05,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:05,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:05,285 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,285 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:05,285 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:05,285 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:05,285 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:05,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,295 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:05,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298a8b12[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:05,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:05,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718405290, max timestamp=1681718405299Num uniques 1136; 2023-04-17 08:00:05,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 08:00:05,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718405290, max timestamp=1681718405299Num uniques 1136; 2023-04-17 08:00:05,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 08:00:05,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681718405290, max timestamp=1681718405299Num uniques 1136; 2023-04-17 08:00:05,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 08:00:05,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:05,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:05,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:05,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:05,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:05,329 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718405326, max timestamp=1681718405330Num uniques 1095; 2023-04-17 08:00:05,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 08:00:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:05,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:05,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718405326, max timestamp=1681718405330Num uniques 1095; 2023-04-17 08:00:05,333 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:05,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:05,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:05,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 08:00:05,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718405326, max timestamp=1681718405330Num uniques 1095; 2023-04-17 08:00:05,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:05,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 08:00:05,333 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:05,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,342 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,365 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,365 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:05,365 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,366 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:05,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:05,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:05,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718405355, max timestamp=1681718405368Num uniques 1054; 2023-04-17 08:00:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-17 08:00:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718405355, max timestamp=1681718405368Num uniques 1054; 2023-04-17 08:00:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-17 08:00:05,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718405355, max timestamp=1681718405368Num uniques 1054; 2023-04-17 08:00:05,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-17 08:00:05,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681718405392, max timestamp=1681718405403Num uniques 1063; 2023-04-17 08:00:05,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-17 08:00:05,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681718405392, max timestamp=1681718405403Num uniques 1063; 2023-04-17 08:00:05,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-17 08:00:05,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681718405392, max timestamp=1681718405404Num uniques 1063; 2023-04-17 08:00:05,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-17 08:00:05,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718405420, max timestamp=1681718405432Num uniques 1022; 2023-04-17 08:00:05,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-17 08:00:05,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718405420, max timestamp=1681718405432Num uniques 1022; 2023-04-17 08:00:05,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-17 08:00:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718405420, max timestamp=1681718405432Num uniques 1022; 2023-04-17 08:00:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-17 08:00:05,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:05,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681718405442, max timestamp=1681718405467Num uniques 1081; 2023-04-17 08:00:05,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681718405442, max timestamp=1681718405467Num uniques 1081; 2023-04-17 08:00:05,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-17 08:00:05,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-17 08:00:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:05,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:05,470 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:05,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:05,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:05,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681718405442, max timestamp=1681718405467Num uniques 1081; 2023-04-17 08:00:05,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-17 08:00:05,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:05,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:05,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:05,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:05,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:05,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:05,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:05,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:05,484 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,485 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:05,485 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:05,490 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,490 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,490 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,490 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:05,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fc26c1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:05,491 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:05,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:05,499 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,499 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:05,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:05,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:05,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681718405494, max timestamp=1681718405499Num uniques 1140; 2023-04-17 08:00:05,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-17 08:00:05,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681718405494, max timestamp=1681718405499Num uniques 1140; 2023-04-17 08:00:05,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-17 08:00:05,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681718405494, max timestamp=1681718405499Num uniques 1140; 2023-04-17 08:00:05,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-17 08:00:05,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:05,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:05,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:05,509 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:05,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:05,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:05,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:05,509 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:05,515 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,515 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,515 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/e4a67b3f8ec04296910deb59278e708b 2023-04-17 08:00:05,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:05,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:05,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,524 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,524 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:05,524 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,524 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,524 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:05,524 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,524 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:05,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/3dca979d7d6140a3acba4d21e796c211 2023-04-17 08:00:05,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681718405540, max timestamp=1681718405547Num uniques 1149; 2023-04-17 08:00:05,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-17 08:00:05,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681718405540, max timestamp=1681718405547Num uniques 1149; 2023-04-17 08:00:05,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-17 08:00:05,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=549, cellSize=25144, totalHeapSize=57336, min timestamp=1681718405527, max timestamp=1681718405547Num uniques 1149; 2023-04-17 08:00:05,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=25144, suffix heap size=147464,new segment heap size=57336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=549 2023-04-17 08:00:05,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718465564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718405562, max timestamp=1681718405567Num uniques 1408; 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=64480, new segment data size=11810, suffix heap size=174400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1408, new segment cells count=258 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718405562, max timestamp=1681718405567Num uniques 1058; 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718405562, max timestamp=1681718405567Num uniques 1058; 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-17 08:00:05,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-17 08:00:05,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718465588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718465589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718465592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681718405584, max timestamp=1681718405591Num uniques 1117; 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681718405584, max timestamp=1681718405591Num uniques 1117; 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681718405584, max timestamp=1681718405591Num uniques 1117; 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-17 08:00:05,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-17 08:00:05,624 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:05,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:05,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:05,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:05,627 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:05,627 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,627 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:05,627 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:05,627 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,628 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,637 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,637 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:05,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:05,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:05,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:05,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:05,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:05,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:05,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,649 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,649 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:05,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e61fb40[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:05,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:05,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,657 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:05,657 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:05,657 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:05,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:05,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:05,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:05,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:05,667 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:05,667 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:05,667 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:05,667 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:05,667 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:05,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,675 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:05,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:05,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:05,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,699 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718465713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8041, totalHeapSize=18544, min timestamp=1681718405701, max timestamp=1681718405723Num uniques 1026; 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681718405701, max timestamp=1681718405723Num uniques 1026; 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46976, new segment data size=8046, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8044, totalHeapSize=18544, min timestamp=1681718405701, max timestamp=1681718405723Num uniques 1026; 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46974, new segment data size=8044, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-17 08:00:05,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46971, new segment data size=8041, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681718405754, max timestamp=1681718405768Num uniques 1035; 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681718405754, max timestamp=1681718405768Num uniques 1035; 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681718405754, max timestamp=1681718405768Num uniques 1035; 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-17 08:00:05,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-17 08:00:05,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:05,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718465796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1681718405792, max timestamp=1681718405796Num uniques 1194; 2023-04-17 08:00:05,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1681718405792, max timestamp=1681718405796Num uniques 1194; 2023-04-17 08:00:05,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=54674, new segment data size=8874, suffix heap size=152144,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=194 2023-04-17 08:00:05,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1681718405792, max timestamp=1681718405796Num uniques 1194; 2023-04-17 08:00:05,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=54674, new segment data size=8874, suffix heap size=152144,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=194 2023-04-17 08:00:05,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=54674, new segment data size=8874, suffix heap size=152144,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=194 2023-04-17 08:00:05,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:05,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:05,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:05,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:05,803 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:05,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:05,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:05,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,812 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:05,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:05,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:05,813 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:05,813 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:05,813 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:05,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:05,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:05,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:05,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:05,823 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:05,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97d2cbe[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:05,825 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:05,826 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,832 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,832 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:05,836 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,836 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:05,836 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:05,836 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:05,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681718405806, max timestamp=1681718405837Num uniques 1053; 2023-04-17 08:00:05,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-17 08:00:05,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681718405806, max timestamp=1681718405837Num uniques 1053; 2023-04-17 08:00:05,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-17 08:00:05,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681718405806, max timestamp=1681718405837Num uniques 1053; 2023-04-17 08:00:05,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-17 08:00:05,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,845 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:05,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:05,848 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:05,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:05,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:05,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:05,848 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:05,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:05,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:05,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,860 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681718405858, max timestamp=1681718405874Num uniques 1162; 2023-04-17 08:00:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-17 08:00:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9700, totalHeapSize=22288, min timestamp=1681718405858, max timestamp=1681718405874Num uniques 1162; 2023-04-17 08:00:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=53210, new segment data size=9700, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-17 08:00:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681718405858, max timestamp=1681718405874Num uniques 1162; 2023-04-17 08:00:05,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:05,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:05,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:05,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:05,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:05,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:05,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:05,896 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:05,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:05,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:05,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718405902, max timestamp=1681718405914Num uniques 1071; 2023-04-17 08:00:05,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-17 08:00:05,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718405902, max timestamp=1681718405914Num uniques 1071; 2023-04-17 08:00:05,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-17 08:00:05,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718405902, max timestamp=1681718405914Num uniques 1071; 2023-04-17 08:00:05,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-17 08:00:05,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/a12038c613b74213a03c74340dbcd1ef 2023-04-17 08:00:05,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/e4a67b3f8ec04296910deb59278e708b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b 2023-04-17 08:00:05,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b, entries=150, sequenceid=889, filesize=11.9 K 2023-04-17 08:00:05,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/3dca979d7d6140a3acba4d21e796c211 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211 2023-04-17 08:00:05,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681718405944, max timestamp=1681718405965Num uniques 1080; 2023-04-17 08:00:05,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-17 08:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681718405944, max timestamp=1681718405961Num uniques 1080; 2023-04-17 08:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681718405944, max timestamp=1681718405961Num uniques 1080; 2023-04-17 08:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-17 08:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-17 08:00:05,996 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:05,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:06,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:06,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211, entries=150, sequenceid=889, filesize=11.9 K 2023-04-17 08:00:06,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:06,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:06,004 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:06,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:06,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:06,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:06,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/a12038c613b74213a03c74340dbcd1ef as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef 2023-04-17 08:00:06,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef, entries=150, sequenceid=889, filesize=11.9 K 2023-04-17 08:00:06,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~31.31 KB/32060, heapSize ~76.34 KB/78176, currentSize=140.89 KB/144270 for 2a1678219a4c91eb2f28c4b713537f8a in 919ms, sequenceid=889, compaction requested=true 2023-04-17 08:00:06,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:06,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:06,014 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:06,014 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:06,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:06,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:06,020 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:06,020 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:06,020 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,021 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8555144290fe4d358ec811e277772d5f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.3 K 2023-04-17 08:00:06,021 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:06,021 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:06,021 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,021 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 8555144290fe4d358ec811e277772d5f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:06,021 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c5870de5c64e4989bedd53c9cddcd1bd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.3 K 2023-04-17 08:00:06,022 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b2768eff2db241c7af915d5db013f335, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1681718402705 2023-04-17 08:00:06,022 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c5870de5c64e4989bedd53c9cddcd1bd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:06,022 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4058db1f34094152981d12568e847f4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1681718402705 2023-04-17 08:00:06,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:06,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:06,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:06,024 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d324f3fb51f0444d904d035823b3e672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681718403340 2023-04-17 08:00:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,025 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting dc33886ab60147168a17cfdd040db55a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681718403340 2023-04-17 08:00:06,025 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e4a67b3f8ec04296910deb59278e708b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,025 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3dca979d7d6140a3acba4d21e796c211, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:06,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:06,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:06,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:06,041 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,041 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:06,041 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=140.89 KB heapSize=393.56 KB 2023-04-17 08:00:06,041 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:06,042 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:06,042 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:06,042 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:06,042 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:06,042 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:06,053 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:06,058 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:06,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:06,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:06,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718466072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718466072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718466074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718466075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,121 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/288936c9860346449e8b8e70accf5e8a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/288936c9860346449e8b8e70accf5e8a 2023-04-17 08:00:06,121 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1251 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/48bda671488a44b59ca37895690826e5 2023-04-17 08:00:06,137 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into 288936c9860346449e8b8e70accf5e8a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:06,138 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:06,138 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=12, startTime=1681718406014; duration=0sec 2023-04-17 08:00:06,138 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:06,138 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:06,139 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:06,139 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:06,139 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,140 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/72e7ccc3b962440c87aef65210556807, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.3 K 2023-04-17 08:00:06,140 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 72e7ccc3b962440c87aef65210556807, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681718401352 2023-04-17 08:00:06,140 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting eae08552615d481ea20c0de64ca0d3b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1681718402705 2023-04-17 08:00:06,141 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e248e479f3a54e1c9319fa1c0c1b1281, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681718403340 2023-04-17 08:00:06,141 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a12038c613b74213a03c74340dbcd1ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:06,157 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:06,176 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1251 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/fcc28de053a94ad49de59376bcc25506 2023-04-17 08:00:06,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718466183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718466187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718466187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718466192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,216 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/e2ff0f140c684f9f883cf7311e3f80d9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e2ff0f140c684f9f883cf7311e3f80d9 2023-04-17 08:00:06,224 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into e2ff0f140c684f9f883cf7311e3f80d9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:06,224 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:06,224 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=12, startTime=1681718406015; duration=0sec 2023-04-17 08:00:06,224 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:06,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718466386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718466393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718466394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718466395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,528 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/45b7f1f39fae48fbb84b4b0d837d0170 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/45b7f1f39fae48fbb84b4b0d837d0170 2023-04-17 08:00:06,536 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into 45b7f1f39fae48fbb84b4b0d837d0170(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:06,536 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:06,537 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=12, startTime=1681718406014; duration=0sec 2023-04-17 08:00:06,537 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:06,630 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1251 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/c13ca4cea2c24d7e8c77eac2717748cf 2023-04-17 08:00:06,638 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/48bda671488a44b59ca37895690826e5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5 2023-04-17 08:00:06,644 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5, entries=150, sequenceid=1251, filesize=11.9 K 2023-04-17 08:00:06,645 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/fcc28de053a94ad49de59376bcc25506 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506 2023-04-17 08:00:06,656 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506, entries=150, sequenceid=1251, filesize=11.9 K 2023-04-17 08:00:06,658 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/c13ca4cea2c24d7e8c77eac2717748cf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf 2023-04-17 08:00:06,664 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf, entries=150, sequenceid=1251, filesize=11.9 K 2023-04-17 08:00:06,664 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~393.52 KB/402960, currentSize=40.25 KB/41220 for 2a1678219a4c91eb2f28c4b713537f8a in 623ms, sequenceid=1251, compaction requested=false 2023-04-17 08:00:06,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:06,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,669 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:06,669 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:06,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:06,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:06,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d2f498[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:06,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:06,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:00:06,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:06,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:06,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:06,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:06,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:06,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:06,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:06,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:06,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:06,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:06,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:06,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:06,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:06,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:06,710 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:06,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:06,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:06,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:06,710 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:06,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:06,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718406709, max timestamp=1681718406730Num uniques 859; 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:06,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:06,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:06,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718406709, max timestamp=1681718406730Num uniques 859; 2023-04-17 08:00:06,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:00:06,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:06,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:06,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:00:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:06,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718406709, max timestamp=1681718406729Num uniques 859; 2023-04-17 08:00:06,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:00:06,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:06,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:06,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:06,733 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:06,733 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718466747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718466747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718466748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718466748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718466750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/8f79b12aee6241a6b93aff5f9ca90e72 2023-04-17 08:00:06,834 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:06,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:06,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:06,842 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:06,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:06,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:06,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:06,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,848 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:06,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:06,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:06,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:06,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:06,850 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,850 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:06,850 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:06,850 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:06,850 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:06,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718466852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718466852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718466853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718466854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:06,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc00eed[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:06,859 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:06,859 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:06,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718466856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:06,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:06,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:06,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:06,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:06,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:06,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:06,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:06,876 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:06,876 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:06,876 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:06,876 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:06,876 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:06,882 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,882 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,882 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:06,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:06,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:06,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:06,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:06,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:06,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:06,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:06,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:06,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:06,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:06,896 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:06,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:06,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:06,996 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:06,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:06,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:06,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:06,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:06,999 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:06,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:06,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:06,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:07,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,007 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:07,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:07,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:07,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:07,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:07,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:07,008 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:07,008 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,008 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:07,008 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:07,009 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,009 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:07,009 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:07,009 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,009 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:07,009 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:07,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:07,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:07,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1a4bce[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:07,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:07,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:07,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:07,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:07,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:07,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:07,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:07,040 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,040 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,042 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,042 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,044 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:07,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:07,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:07,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:07,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:07,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718467058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718467058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718467059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718467058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718467062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:07,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:07,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:07,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:07,159 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:07,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:07,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:07,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/935484a1936a40fe911e9e47aafdea6b 2023-04-17 08:00:07,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:07,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:07,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:07,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:07,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:07,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:07,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:07,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:07,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:07,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:07,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,207 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,207 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5e95cd2ecd064ac2b1b490b9c0097b59 2023-04-17 08:00:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:07,210 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:07,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d62860f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:07,210 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/8f79b12aee6241a6b93aff5f9ca90e72 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72 2023-04-17 08:00:07,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:07,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:07,221 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:07,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:07,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72, entries=100, sequenceid=1265, filesize=9.5 K 2023-04-17 08:00:07,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/935484a1936a40fe911e9e47aafdea6b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b 2023-04-17 08:00:07,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:07,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:07,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:07,235 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:07,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:07,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:07,235 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:07,235 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:07,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b, entries=100, sequenceid=1265, filesize=9.5 K 2023-04-17 08:00:07,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/5e95cd2ecd064ac2b1b490b9c0097b59 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59 2023-04-17 08:00:07,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59, entries=100, sequenceid=1265, filesize=9.5 K 2023-04-17 08:00:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 2a1678219a4c91eb2f28c4b713537f8a in 550ms, sequenceid=1265, compaction requested=true 2023-04-17 08:00:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,246 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:07,247 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:07,247 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,248 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:07,248 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,248 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,248 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/288936c9860346449e8b8e70accf5e8a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=34.2 K 2023-04-17 08:00:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,249 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:07,249 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:07,249 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,249 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/45b7f1f39fae48fbb84b4b0d837d0170, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=34.2 K 2023-04-17 08:00:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,250 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 288936c9860346449e8b8e70accf5e8a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:07,250 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 45b7f1f39fae48fbb84b4b0d837d0170, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:07,250 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 48bda671488a44b59ca37895690826e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1251, earliestPutTs=1681718405986 2023-04-17 08:00:07,250 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fcc28de053a94ad49de59376bcc25506, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1251, earliestPutTs=1681718405986 2023-04-17 08:00:07,251 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 8f79b12aee6241a6b93aff5f9ca90e72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406695 2023-04-17 08:00:07,251 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 935484a1936a40fe911e9e47aafdea6b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406695 2023-04-17 08:00:07,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:07,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:07,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:07,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,258 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:07,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:07,257 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,258 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,263 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:07,272 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#88 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:07,293 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/3c11a7a6cb0d4c1c900b7d04edb21489 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3c11a7a6cb0d4c1c900b7d04edb21489 2023-04-17 08:00:07,300 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into 3c11a7a6cb0d4c1c900b7d04edb21489(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:07,300 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,300 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=13, startTime=1681718407247; duration=0sec 2023-04-17 08:00:07,301 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:07,301 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:07,303 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:07,303 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:07,303 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,303 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e2ff0f140c684f9f883cf7311e3f80d9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=34.2 K 2023-04-17 08:00:07,303 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e2ff0f140c684f9f883cf7311e3f80d9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681718404469 2023-04-17 08:00:07,304 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c13ca4cea2c24d7e8c77eac2717748cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1251, earliestPutTs=1681718405986 2023-04-17 08:00:07,305 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5e95cd2ecd064ac2b1b490b9c0097b59, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406695 2023-04-17 08:00:07,311 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:07,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:07,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:07,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:07,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:07,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:07,361 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:07,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:07,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:07,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:07,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-17 08:00:07,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:07,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:07,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:07,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:07,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:07,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:07,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:07,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:07,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:07,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:07,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:07,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:07,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718467374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:07,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:07,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:07,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:07,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:07,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:07,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718467376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718467376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718467377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718467378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:07,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:07,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,399 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,399 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:07,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e77c10f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:07,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:07,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,407 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:07,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:07,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:07,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:07,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1303 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/478b7bb6517d4095af3d09a9a0d94933 2023-04-17 08:00:07,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:07,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:07,418 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:07,418 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:07,418 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:07,418 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:07,418 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:07,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1303 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/95117d2e29c4449fb0208a1553b7bba6 2023-04-17 08:00:07,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,446 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:07,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,446 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:07,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:07,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1303 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/dbf03e2e7e994c2685982b2e89f56295 2023-04-17 08:00:07,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/478b7bb6517d4095af3d09a9a0d94933 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933 2023-04-17 08:00:07,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933, entries=200, sequenceid=1303, filesize=14.3 K 2023-04-17 08:00:07,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718467479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/95117d2e29c4449fb0208a1553b7bba6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6 2023-04-17 08:00:07,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718467483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718467484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718467485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718467485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6, entries=150, sequenceid=1303, filesize=11.9 K 2023-04-17 08:00:07,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/dbf03e2e7e994c2685982b2e89f56295 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295 2023-04-17 08:00:07,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295, entries=150, sequenceid=1303, filesize=11.9 K 2023-04-17 08:00:07,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for 2a1678219a4c91eb2f28c4b713537f8a in 140ms, sequenceid=1303, compaction requested=false 2023-04-17 08:00:07,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,546 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:07,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:07,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:07,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:07,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:07,550 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:07,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:07,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:07,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:07,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:07,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:07,558 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:07,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:07,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:07,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:07,559 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:07,560 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,580 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b1fcf3b034894bba9366d2fddf8befad 2023-04-17 08:00:07,595 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/981b76cc96f94690997f0333f1b41dce 2023-04-17 08:00:07,620 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/086fb90b18844fe8849a2ff8ca8a3a11 2023-04-17 08:00:07,628 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b1fcf3b034894bba9366d2fddf8befad as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad 2023-04-17 08:00:07,637 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad, entries=100, sequenceid=1312, filesize=9.5 K 2023-04-17 08:00:07,638 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/981b76cc96f94690997f0333f1b41dce as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce 2023-04-17 08:00:07,653 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce, entries=100, sequenceid=1312, filesize=9.5 K 2023-04-17 08:00:07,656 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/086fb90b18844fe8849a2ff8ca8a3a11 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11 2023-04-17 08:00:07,662 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11, entries=100, sequenceid=1312, filesize=9.5 K 2023-04-17 08:00:07,663 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 2a1678219a4c91eb2f28c4b713537f8a in 104ms, sequenceid=1312, compaction requested=true 2023-04-17 08:00:07,663 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,664 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:07,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:07,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,668 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:07,668 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:07,682 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,682 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,683 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,683 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:07,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62ec194a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:07,686 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:07,686 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:07,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:07,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:07,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:07,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:07,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:07,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:07,701 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/dcd229a7f6644eac89adbf464340a58f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/dcd229a7f6644eac89adbf464340a58f 2023-04-17 08:00:07,711 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into dcd229a7f6644eac89adbf464340a58f(size=12.9 K), total size for store is 36.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:07,711 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,711 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=13, startTime=1681718407247; duration=0sec 2023-04-17 08:00:07,712 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:07,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:07,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:07,718 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:07,718 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:07,719 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:07,719 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:07,719 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:07,734 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/f402f80c396347318653bb2b1bf3c338 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f402f80c396347318653bb2b1bf3c338 2023-04-17 08:00:07,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/91b26429a49942a0a844389462f997fe 2023-04-17 08:00:07,746 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into f402f80c396347318653bb2b1bf3c338(size=12.9 K), total size for store is 34.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:07,746 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:07,746 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=13, startTime=1681718407247; duration=0sec 2023-04-17 08:00:07,746 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:07,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718467742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718407712, max timestamp=1681718407748Num uniques 859; 2023-04-17 08:00:07,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718407712, max timestamp=1681718407748Num uniques 859; 2023-04-17 08:00:07,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718407712, max timestamp=1681718407748Num uniques 859; 2023-04-17 08:00:07,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 08:00:07,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 08:00:07,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 08:00:07,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718467780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718467780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718467781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718467786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,840 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,840 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:07,840 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,844 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:07,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:07,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718467852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:07,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:07,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:07,874 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,874 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:07,874 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,874 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,874 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:07,874 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,874 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:07,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:07,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718467887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718467887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718467888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:07,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718467891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,974 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:07,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:07,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:07,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:07,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:07,979 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:07,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:07,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:07,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,990 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:07,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:07,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:07,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:07,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:07,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:07,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:07,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:07,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:07,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:07,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:07,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:07,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:07,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,027 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,027 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,027 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,027 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d376ae2[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,050 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,051 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,051 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,051 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718468056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,060 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,060 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,060 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,061 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,061 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,082 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,082 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,082 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,085 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,085 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718468092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718468092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718468092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718468094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/e7e851db16de4c2d85267eb7f33221a9 2023-04-17 08:00:08,182 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:08,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:08,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:08,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:08,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:08,185 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:08,185 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,185 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:08,185 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:08,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/173c58cea9c64dc58f2782dc4d4530d8 2023-04-17 08:00:08,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/91b26429a49942a0a844389462f997fe as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe 2023-04-17 08:00:08,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:08,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe, entries=150, sequenceid=1323, filesize=11.9 K 2023-04-17 08:00:08,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:08,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:08,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:08,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:08,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:08,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:08,200 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,200 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:08,201 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:08,201 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:08,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:08,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/e7e851db16de4c2d85267eb7f33221a9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9 2023-04-17 08:00:08,202 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:08,202 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9, entries=150, sequenceid=1323, filesize=11.9 K 2023-04-17 08:00:08,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/173c58cea9c64dc58f2782dc4d4530d8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8 2023-04-17 08:00:08,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8, entries=150, sequenceid=1323, filesize=11.9 K 2023-04-17 08:00:08,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for 2a1678219a4c91eb2f28c4b713537f8a in 517ms, sequenceid=1323, compaction requested=true 2023-04-17 08:00:08,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:08,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:08,212 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:08,212 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:08,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:08,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,214 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,214 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:08,214 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49728 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:08,214 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,214 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:08,214 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,214 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3c11a7a6cb0d4c1c900b7d04edb21489, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=46.2 K 2023-04-17 08:00:08,214 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/dcd229a7f6644eac89adbf464340a58f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.6 K 2023-04-17 08:00:08,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445cd9b5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,215 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3c11a7a6cb0d4c1c900b7d04edb21489, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406005 2023-04-17 08:00:08,215 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting dcd229a7f6644eac89adbf464340a58f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406005 2023-04-17 08:00:08,215 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 95117d2e29c4449fb0208a1553b7bba6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1303, earliestPutTs=1681718406745 2023-04-17 08:00:08,216 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 478b7bb6517d4095af3d09a9a0d94933, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1303, earliestPutTs=1681718406745 2023-04-17 08:00:08,216 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 981b76cc96f94690997f0333f1b41dce, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1681718407375 2023-04-17 08:00:08,216 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b1fcf3b034894bba9366d2fddf8befad, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1681718407375 2023-04-17 08:00:08,216 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e7e851db16de4c2d85267eb7f33221a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1681718407692 2023-04-17 08:00:08,216 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 91b26429a49942a0a844389462f997fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1681718407692 2023-04-17 08:00:08,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,224 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,224 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:08,225 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:08,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,238 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,238 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,240 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/c461299edfa148d9bfcdb759efbbff5b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c461299edfa148d9bfcdb759efbbff5b 2023-04-17 08:00:08,245 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into c461299edfa148d9bfcdb759efbbff5b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:08,245 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:08,245 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=12, startTime=1681718408212; duration=0sec 2023-04-17 08:00:08,245 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:08,245 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:08,247 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:08,247 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/C is initiating minor compaction (all files) 2023-04-17 08:00:08,247 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/C in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,248 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f402f80c396347318653bb2b1bf3c338, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=46.2 K 2023-04-17 08:00:08,248 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f402f80c396347318653bb2b1bf3c338, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1681718406005 2023-04-17 08:00:08,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,249 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting dbf03e2e7e994c2685982b2e89f56295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1303, earliestPutTs=1681718406745 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,251 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 086fb90b18844fe8849a2ff8ca8a3a11, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1681718407375 2023-04-17 08:00:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,251 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 173c58cea9c64dc58f2782dc4d4530d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1681718407692 2023-04-17 08:00:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,258 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#C#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:08,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:08,361 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=132.99 KB heapSize=382.24 KB 2023-04-17 08:00:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:08,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:08,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:08,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:08,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:08,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:08,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:08,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:08,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:08,366 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:08,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:08,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:08,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:08,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:08,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:08,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:08,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:08,375 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:08,375 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,382 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718468377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d30788[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,391 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,391 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,391 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,399 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718468397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718468399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718468401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718468402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,415 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,416 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,424 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,424 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,424 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,424 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,424 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,424 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718468483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,524 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:08,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:08,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:08,527 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:08,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:08,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:08,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:08,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:08,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:08,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:08,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:08,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:08,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:08,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:08,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:08,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:08,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:08,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:08,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,549 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,549 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f76e11[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,586 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,586 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,586 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,586 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,588 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,589 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,607 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,607 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,643 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/e2f54f287d56481a9f21a5ed41efc4f9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e2f54f287d56481a9f21a5ed41efc4f9 2023-04-17 08:00:08,648 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into e2f54f287d56481a9f21a5ed41efc4f9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:08,649 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:08,649 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=12, startTime=1681718408212; duration=0sec 2023-04-17 08:00:08,649 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:08,672 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/b313f079abde4d449ddbb534231cda79 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b313f079abde4d449ddbb534231cda79 2023-04-17 08:00:08,677 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/C of 2a1678219a4c91eb2f28c4b713537f8a into b313f079abde4d449ddbb534231cda79(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:08,677 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:08,677 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/C, priority=12, startTime=1681718408212; duration=0sec 2023-04-17 08:00:08,677 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:08,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718468687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,707 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:08,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:08,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:08,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:08,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:08,710 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:08,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:08,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:08,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,710 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:08,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:08,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:08,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:08,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:08,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:08,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,732 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d17257[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,740 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,740 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,740 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,748 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,750 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,750 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,757 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,759 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,771 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=1363 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b63f1015e8a347ea8ce2b70d6909f834 2023-04-17 08:00:08,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:08,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:08,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:08,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:08,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:08,874 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:08,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:08,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:08,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,874 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,882 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:08,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:08,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:08,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:08,883 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,883 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:08,883 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:08,883 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:08,883 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:08,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:08,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52629c89[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:08,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:08,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:08,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:08,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:08,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:08,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718468904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:08,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:08,907 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718468905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718468906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718468908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:08,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:08,916 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:08,917 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:08,917 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:08,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:08,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:08,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:08,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:08,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:08,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:08,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:08,932 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:08,932 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:08,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:08,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:08,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718468990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,032 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:09,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:09,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:09,035 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:09,035 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,035 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:09,035 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:09,036 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,036 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,045 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:09,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:09,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:09,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:09,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:09,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:09,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:09,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:09,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e213e0a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:09,058 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:09,058 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:09,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:09,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:09,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:09,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:09,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:09,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:09,075 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:09,076 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:09,076 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:09,076 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:09,076 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:09,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:09,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,147 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:09,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:09,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,148 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=1363 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/9647f17d07be452393401cd272d0e127 2023-04-17 08:00:09,248 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:09,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:09,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:09,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:09,251 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:09,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:09,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:09,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:09,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:09,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:09,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:09,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:09,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:09,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:09,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:09,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:09,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:09,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,274 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,274 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:09,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732b4039[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:09,275 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:09,275 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,282 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,282 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:09,282 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:09,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:09,282 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:09,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:09,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:09,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:09,292 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:09,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:09,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:09,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:09,292 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:09,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,300 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:09,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:09,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:09,307 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:09,307 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,407 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:09,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:09,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:09,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:09,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:09,410 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:09,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:09,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:09,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:09,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:09,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:09,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:09,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:09,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:09,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:09,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,432 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:09,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@893c7c1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:09,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:09,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:09,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:09,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:09,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:09,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:09,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:09,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:09,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:09,451 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:09,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:09,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:09,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:09,451 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:09,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,458 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:09,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:09,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,466 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:09,466 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:09,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:09,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718469493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,566 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:09,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:09,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:09,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:09,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:09,569 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:09,569 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,569 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:09,569 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:09,569 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,569 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:09,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:09,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:09,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:09,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:09,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:09,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:09,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:09,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:09,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e53848e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:09,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:09,592 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=1363 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/ef0fa4b184c04e3aaabdc049b3401e64 2023-04-17 08:00:09,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:09,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:09,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:09,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:09,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/b63f1015e8a347ea8ce2b70d6909f834 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834 2023-04-17 08:00:09,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834, entries=200, sequenceid=1363, filesize=14.3 K 2023-04-17 08:00:09,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/9647f17d07be452393401cd272d0e127 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127 2023-04-17 08:00:09,607 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:09,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:09,609 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:09,609 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:09,609 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:09,609 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:09,609 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:09,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127, entries=150, sequenceid=1363, filesize=11.9 K 2023-04-17 08:00:09,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/ef0fa4b184c04e3aaabdc049b3401e64 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef0fa4b184c04e3aaabdc049b3401e64 2023-04-17 08:00:09,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:09,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:09,616 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef0fa4b184c04e3aaabdc049b3401e64, entries=150, sequenceid=1363, filesize=11.9 K 2023-04-17 08:00:09,617 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:09,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~132.99 KB/136185, heapSize ~382.20 KB/391368, currentSize=46.96 KB/48090 for 2a1678219a4c91eb2f28c4b713537f8a in 1257ms, sequenceid=1363, compaction requested=false 2023-04-17 08:00:09,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:09,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:09,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:09,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:09,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:09,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:09,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:09,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:09,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:09,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:09,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:09,732 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:09,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:09,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:09,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:09,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:09,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:09,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:09,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:09,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:09,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:09,741 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:09,742 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:09,742 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. started... 2023-04-17 08:00:09,742 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:09,742 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:09,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:09,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. as already flushing 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681718409922, max timestamp=1681718409928Num uniques 859; 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681718409922, max timestamp=1681718409928Num uniques 859; 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681718409922, max timestamp=1681718409928Num uniques 859; 2023-04-17 08:00:09,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-17 08:00:09,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718469938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718469939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718469939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:09,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718469940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718470041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718470043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718470044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718470044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,150 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1375 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d6d9976b11944ddda854e13914461471 2023-04-17 08:00:10,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718470244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718470247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718470248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718470248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58318 deadline: 1681718470499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58376 deadline: 1681718470549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58330 deadline: 1681718470553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58340 deadline: 1681718470555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:10,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:58356 deadline: 1681718470555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:10,562 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1375 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/af7a8be5b64344949f1ca0853c8e1dde 2023-04-17 08:00:10,791 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e044e22 to 127.0.0.1:62477 2023-04-17 08:00:10,791 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:10,792 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:62477 2023-04-17 08:00:10,792 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:10,795 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18f7ec13 to 127.0.0.1:62477 2023-04-17 08:00:10,795 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:10,798 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ab9467e to 127.0.0.1:62477 2023-04-17 08:00:10,798 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:10,973 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1375 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/6ddf163732d847929f58d678f66ab61c 2023-04-17 08:00:10,984 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/d6d9976b11944ddda854e13914461471 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471 2023-04-17 08:00:10,989 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471, entries=150, sequenceid=1375, filesize=11.9 K 2023-04-17 08:00:10,990 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/af7a8be5b64344949f1ca0853c8e1dde as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde 2023-04-17 08:00:10,996 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde, entries=150, sequenceid=1375, filesize=11.9 K 2023-04-17 08:00:10,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/6ddf163732d847929f58d678f66ab61c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6ddf163732d847929f58d678f66ab61c 2023-04-17 08:00:11,002 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6ddf163732d847929f58d678f66ab61c, entries=150, sequenceid=1375, filesize=11.9 K 2023-04-17 08:00:11,003 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2a1678219a4c91eb2f28c4b713537f8a in 1261ms, sequenceid=1375, compaction requested=true 2023-04-17 08:00:11,003 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:11,003 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:11,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:11,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:11,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:11,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:11,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:11,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-17 08:00:11,058 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:62477 2023-04-17 08:00:11,058 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:11,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:11,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:11,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:11,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:11,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:11,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:11,061 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ecabb06 to 127.0.0.1:62477 2023-04-17 08:00:11,061 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:11,063 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45864842 to 127.0.0.1:62477 2023-04-17 08:00:11,063 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:11,064 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:62477 2023-04-17 08:00:11,064 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:11,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:11,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:11,158 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:11,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:11,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39752220[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:11,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:11,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,183 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:11,183 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:11,183 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:11,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:11,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:11,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:11,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:11,196 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:11,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:11,203 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:11,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:11,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:11,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:11,204 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:11,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,215 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,217 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:11,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:11,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:11,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:11,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:11,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:11,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:11,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:11,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:11,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:11,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:11,233 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:11,233 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:11,233 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,233 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:11,233 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:11,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:11,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:11,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1410 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/a6ed33a893c346c6a4194161fbeac424 2023-04-17 08:00:11,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1410 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/c025710c9ec34b8a8d0f5ac0add07794 2023-04-17 08:00:12,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1410 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/397c816d643a4ff38d8ab8fa969e2428 2023-04-17 08:00:12,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/a6ed33a893c346c6a4194161fbeac424 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424 2023-04-17 08:00:12,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424, entries=150, sequenceid=1410, filesize=11.9 K 2023-04-17 08:00:12,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/c025710c9ec34b8a8d0f5ac0add07794 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794 2023-04-17 08:00:12,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794, entries=150, sequenceid=1410, filesize=11.9 K 2023-04-17 08:00:12,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/397c816d643a4ff38d8ab8fa969e2428 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/397c816d643a4ff38d8ab8fa969e2428 2023-04-17 08:00:12,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/397c816d643a4ff38d8ab8fa969e2428, entries=150, sequenceid=1410, filesize=11.9 K 2023-04-17 08:00:12,348 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=20.13 KB/20610 for 2a1678219a4c91eb2f28c4b713537f8a in 1290ms, sequenceid=1410, compaction requested=true 2023-04-17 08:00:12,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:12,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:12,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:12,348 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:12,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:12,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:12,349 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52308 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:12,349 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/A is initiating minor compaction (all files) 2023-04-17 08:00:12,349 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:12,349 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/A in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,350 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 2a1678219a4c91eb2f28c4b713537f8a/B is initiating minor compaction (all files) 2023-04-17 08:00:12,350 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a1678219a4c91eb2f28c4b713537f8a/B in TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,350 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c461299edfa148d9bfcdb759efbbff5b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=51.1 K 2023-04-17 08:00:12,350 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e2f54f287d56481a9f21a5ed41efc4f9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp, totalSize=48.7 K 2023-04-17 08:00:12,350 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c461299edfa148d9bfcdb759efbbff5b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1681718407692 2023-04-17 08:00:12,350 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e2f54f287d56481a9f21a5ed41efc4f9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1681718407692 2023-04-17 08:00:12,350 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b63f1015e8a347ea8ce2b70d6909f834, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1363, earliestPutTs=1681718407774 2023-04-17 08:00:12,350 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9647f17d07be452393401cd272d0e127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1363, earliestPutTs=1681718407780 2023-04-17 08:00:12,351 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d6d9976b11944ddda854e13914461471, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1375, earliestPutTs=1681718408369 2023-04-17 08:00:12,351 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting af7a8be5b64344949f1ca0853c8e1dde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1375, earliestPutTs=1681718408369 2023-04-17 08:00:12,351 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a6ed33a893c346c6a4194161fbeac424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1410, earliestPutTs=1681718409934 2023-04-17 08:00:12,351 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting c025710c9ec34b8a8d0f5ac0add07794, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1410, earliestPutTs=1681718409934 2023-04-17 08:00:12,367 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#A#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:12,370 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a1678219a4c91eb2f28c4b713537f8a#B#compaction#112 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:12,510 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ca212c to 127.0.0.1:62477 2023-04-17 08:00:12,511 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 309 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 268 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 325 2023-04-17 08:00:12,511 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 247 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 4676 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 4511 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2030 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 6088 rows 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2017 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 6047 rows 2023-04-17 08:00:12,512 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:00:12,512 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490ff358 to 127.0.0.1:62477 2023-04-17 08:00:12,513 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:12,519 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:00:12,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:00:12,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:12,533 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718412533"}]},"ts":"1681718412533"} 2023-04-17 08:00:12,534 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:00:12,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:12,562 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:00:12,564 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, UNASSIGN}] 2023-04-17 08:00:12,565 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, UNASSIGN 2023-04-17 08:00:12,566 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2a1678219a4c91eb2f28c4b713537f8a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:12,566 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718412566"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718412566"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718412566"}]},"ts":"1681718412566"} 2023-04-17 08:00:12,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:12,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:12,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:12,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 2a1678219a4c91eb2f28c4b713537f8a, disabling compactions & flushes 2023-04-17 08:00:12,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,786 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/f52edcc3f6ce4425b2022fd95faac305 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/f52edcc3f6ce4425b2022fd95faac305 2023-04-17 08:00:12,787 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/416b94195a694670a353ff90bc17e20f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/416b94195a694670a353ff90bc17e20f 2023-04-17 08:00:12,792 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/A of 2a1678219a4c91eb2f28c4b713537f8a into f52edcc3f6ce4425b2022fd95faac305(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:12,792 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a1678219a4c91eb2f28c4b713537f8a/B of 2a1678219a4c91eb2f28c4b713537f8a into 416b94195a694670a353ff90bc17e20f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:12,792 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:12,792 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:12,792 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/A, priority=12, startTime=1681718412348; duration=0sec 2023-04-17 08:00:12,792 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a., storeName=2a1678219a4c91eb2f28c4b713537f8a/B, priority=12, startTime=1681718412348; duration=0sec 2023-04-17 08:00:12,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,792 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:12,792 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:12,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,792 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. because compaction request was cancelled 2023-04-17 08:00:12,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. after waiting 0 ms 2023-04-17 08:00:12,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:12,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 2a1678219a4c91eb2f28c4b713537f8a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=A 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=B 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a1678219a4c91eb2f28c4b713537f8a, store=C 2023-04-17 08:00:12,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:12,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:13,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:13,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1419 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/013bc24c636a4b81872d8b05943a757b 2023-04-17 08:00:13,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1419 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/d7fd715dd5ed4b24bb59880fc6cadf3c 2023-04-17 08:00:13,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:14,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1419 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/b15267bec63f42cab5152cf994eb65ab 2023-04-17 08:00:14,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/A/013bc24c636a4b81872d8b05943a757b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/013bc24c636a4b81872d8b05943a757b 2023-04-17 08:00:14,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/013bc24c636a4b81872d8b05943a757b, entries=50, sequenceid=1419, filesize=7.1 K 2023-04-17 08:00:14,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/B/d7fd715dd5ed4b24bb59880fc6cadf3c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7fd715dd5ed4b24bb59880fc6cadf3c 2023-04-17 08:00:14,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7fd715dd5ed4b24bb59880fc6cadf3c, entries=50, sequenceid=1419, filesize=7.1 K 2023-04-17 08:00:14,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/.tmp/C/b15267bec63f42cab5152cf994eb65ab as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b15267bec63f42cab5152cf994eb65ab 2023-04-17 08:00:14,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b15267bec63f42cab5152cf994eb65ab, entries=50, sequenceid=1419, filesize=7.1 K 2023-04-17 08:00:14,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2a1678219a4c91eb2f28c4b713537f8a in 1272ms, sequenceid=1419, compaction requested=true 2023-04-17 08:00:14,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ada35bd9d03f4ebb9904198f8609f6b5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0f0f12938d724529b09edbaec2446f65, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/52ad8118c08043c6844ccd5d6484e3d5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/7dbd4c0a887446f7a3b76eb8b99d3863, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a06944de1e1846aaabc9e090064e7080, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/4e4217fe19f24a95bb964718e71b9c59, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8555144290fe4d358ec811e277772d5f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/288936c9860346449e8b8e70accf5e8a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/dcd229a7f6644eac89adbf464340a58f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c461299edfa148d9bfcdb759efbbff5b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424] to archive 2023-04-17 08:00:14,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:14,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/6f1f6e9540184054857e1ac14317eb7b 2023-04-17 08:00:14,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a8721e52aead42fe99ca3649a63dcae3 2023-04-17 08:00:14,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ada35bd9d03f4ebb9904198f8609f6b5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ada35bd9d03f4ebb9904198f8609f6b5 2023-04-17 08:00:14,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/ed83bcbf5f634089a97099de8f5cfaaf 2023-04-17 08:00:14,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/340e457151664f4b8d9f5e9878533874 2023-04-17 08:00:14,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0075b27d356a45ae9859acf4689f6689 2023-04-17 08:00:14,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0f0f12938d724529b09edbaec2446f65 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/0f0f12938d724529b09edbaec2446f65 2023-04-17 08:00:14,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/da5821a905ef4e0c943ab157da637e50 2023-04-17 08:00:14,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3d01738eabfd4d8a9f24537db00ae79b 2023-04-17 08:00:14,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/3177b3e8d1fc43329a8ca83772148510 2023-04-17 08:00:14,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/52ad8118c08043c6844ccd5d6484e3d5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/52ad8118c08043c6844ccd5d6484e3d5 2023-04-17 08:00:14,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c127700a53e8453fad86d48e0d96d33e 2023-04-17 08:00:14,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d91b3874ab964c31b03fee07f2382ecb 2023-04-17 08:00:14,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/7dbd4c0a887446f7a3b76eb8b99d3863 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/7dbd4c0a887446f7a3b76eb8b99d3863 2023-04-17 08:00:14,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/13cfbe2ff23f49ddb0acbf20099efe42 2023-04-17 08:00:14,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5e41ecf8db374dc994d443fdddbd74c7 2023-04-17 08:00:14,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a06944de1e1846aaabc9e090064e7080 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a06944de1e1846aaabc9e090064e7080 2023-04-17 08:00:14,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/5ef05d4c2282468da808a0d83c96da60 2023-04-17 08:00:14,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/99d5cf44663e4fae848df4093fcea496 2023-04-17 08:00:14,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/4e4217fe19f24a95bb964718e71b9c59 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/4e4217fe19f24a95bb964718e71b9c59 2023-04-17 08:00:14,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/90ca1e1d5c6440f19e42699f4eaeef54 2023-04-17 08:00:14,115 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8555144290fe4d358ec811e277772d5f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8555144290fe4d358ec811e277772d5f 2023-04-17 08:00:14,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/edf0ba5e5e204947b6795b8060bc6c12 2023-04-17 08:00:14,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b2768eff2db241c7af915d5db013f335 2023-04-17 08:00:14,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d324f3fb51f0444d904d035823b3e672 2023-04-17 08:00:14,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/288936c9860346449e8b8e70accf5e8a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/288936c9860346449e8b8e70accf5e8a 2023-04-17 08:00:14,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/e4a67b3f8ec04296910deb59278e708b 2023-04-17 08:00:14,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/48bda671488a44b59ca37895690826e5 2023-04-17 08:00:14,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/dcd229a7f6644eac89adbf464340a58f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/dcd229a7f6644eac89adbf464340a58f 2023-04-17 08:00:14,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/8f79b12aee6241a6b93aff5f9ca90e72 2023-04-17 08:00:14,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/478b7bb6517d4095af3d09a9a0d94933 2023-04-17 08:00:14,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b1fcf3b034894bba9366d2fddf8befad 2023-04-17 08:00:14,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c461299edfa148d9bfcdb759efbbff5b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/c461299edfa148d9bfcdb759efbbff5b 2023-04-17 08:00:14,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/91b26429a49942a0a844389462f997fe 2023-04-17 08:00:14,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/b63f1015e8a347ea8ce2b70d6909f834 2023-04-17 08:00:14,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/d6d9976b11944ddda854e13914461471 2023-04-17 08:00:14,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/a6ed33a893c346c6a4194161fbeac424 2023-04-17 08:00:14,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ff007bdc7f584c4e89c92f897d211311, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ec7f9684da614d219e5153cd1e4d49e4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/37fb56c00aff4c61af921c9727547a33, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/afedf69be11c47daa2cd3b957c97e544, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d0d46495871f4d7d935630a81bbb3a9b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7d391b8643f48c4b2920c2c05337057, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c5870de5c64e4989bedd53c9cddcd1bd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/45b7f1f39fae48fbb84b4b0d837d0170, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3c11a7a6cb0d4c1c900b7d04edb21489, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e2f54f287d56481a9f21a5ed41efc4f9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794] to archive 2023-04-17 08:00:14,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:14,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/625341aeb25f448f960d666705da3621 2023-04-17 08:00:14,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2c091b60c88c40a78605be0f48c2a126 2023-04-17 08:00:14,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ff007bdc7f584c4e89c92f897d211311 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ff007bdc7f584c4e89c92f897d211311 2023-04-17 08:00:14,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/1ba39d27fad3418b98b5cc23eeb99e90 2023-04-17 08:00:14,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/f9c32efaa0ef4149a2cfae714d6c2ac3 2023-04-17 08:00:14,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2d4345c17c7148a6b522cbe4f78acb70 2023-04-17 08:00:14,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ec7f9684da614d219e5153cd1e4d49e4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ec7f9684da614d219e5153cd1e4d49e4 2023-04-17 08:00:14,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/6114fca8d72a4bffb2c83b434053e324 2023-04-17 08:00:14,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/a88b5acb51964bb2be69c5fb12401de0 2023-04-17 08:00:14,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/37fb56c00aff4c61af921c9727547a33 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/37fb56c00aff4c61af921c9727547a33 2023-04-17 08:00:14,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/0b4ccca125414363809988c5ce15f99d 2023-04-17 08:00:14,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/62309854370646f1a86bc7e404c3de30 2023-04-17 08:00:14,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/afedf69be11c47daa2cd3b957c97e544 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/afedf69be11c47daa2cd3b957c97e544 2023-04-17 08:00:14,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/09f2b3da60404582907df3142027d080 2023-04-17 08:00:14,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2a6fe5182c5c46b79ab01fee0095b9c4 2023-04-17 08:00:14,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d0d46495871f4d7d935630a81bbb3a9b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d0d46495871f4d7d935630a81bbb3a9b 2023-04-17 08:00:14,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/899eb9adb4ff4574867026c11e9eb59c 2023-04-17 08:00:14,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/ae8d22914be84e14a51403ccdc698e28 2023-04-17 08:00:14,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7d391b8643f48c4b2920c2c05337057 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7d391b8643f48c4b2920c2c05337057 2023-04-17 08:00:14,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/2cddeaf6f70e48428b91296064adee8e 2023-04-17 08:00:14,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/365f6a46d4f94c0cadb67454557f1a78 2023-04-17 08:00:14,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c5870de5c64e4989bedd53c9cddcd1bd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c5870de5c64e4989bedd53c9cddcd1bd 2023-04-17 08:00:14,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/92c1d8f97416405896b7dd1821e50d6c 2023-04-17 08:00:14,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/4058db1f34094152981d12568e847f4c 2023-04-17 08:00:14,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/dc33886ab60147168a17cfdd040db55a 2023-04-17 08:00:14,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/45b7f1f39fae48fbb84b4b0d837d0170 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/45b7f1f39fae48fbb84b4b0d837d0170 2023-04-17 08:00:14,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3dca979d7d6140a3acba4d21e796c211 2023-04-17 08:00:14,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/fcc28de053a94ad49de59376bcc25506 2023-04-17 08:00:14,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3c11a7a6cb0d4c1c900b7d04edb21489 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/3c11a7a6cb0d4c1c900b7d04edb21489 2023-04-17 08:00:14,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/935484a1936a40fe911e9e47aafdea6b 2023-04-17 08:00:14,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/95117d2e29c4449fb0208a1553b7bba6 2023-04-17 08:00:14,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/981b76cc96f94690997f0333f1b41dce 2023-04-17 08:00:14,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e2f54f287d56481a9f21a5ed41efc4f9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e2f54f287d56481a9f21a5ed41efc4f9 2023-04-17 08:00:14,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/e7e851db16de4c2d85267eb7f33221a9 2023-04-17 08:00:14,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/9647f17d07be452393401cd272d0e127 2023-04-17 08:00:14,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/af7a8be5b64344949f1ca0853c8e1dde 2023-04-17 08:00:14,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/c025710c9ec34b8a8d0f5ac0add07794 2023-04-17 08:00:14,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/29303d82939940cfbef0f0cc05093363, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c6cff3b039b64056b20612a1f4597f0b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/11dcecf9eba24bbdbb7343af9d4bdeb1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/26e74304e023480e99e72c4f61a66a5e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2f0ce7dfff814c40921d06c73aed3404, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6f7a263d6c2540a79f5bfa0e993c9bfc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/72e7ccc3b962440c87aef65210556807, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e2ff0f140c684f9f883cf7311e3f80d9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f402f80c396347318653bb2b1bf3c338, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8] to archive 2023-04-17 08:00:14,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:14,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6a5966b5caba49999602c0f9f1a75690 2023-04-17 08:00:14,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e4a0de06c24a46b083fddf5d18cffe84 2023-04-17 08:00:14,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/29303d82939940cfbef0f0cc05093363 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/29303d82939940cfbef0f0cc05093363 2023-04-17 08:00:14,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e8fb8f0bb97b4b77acadb2a03bf3d732 2023-04-17 08:00:14,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c0f97477f09e4d08985191c597ce8989 2023-04-17 08:00:14,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b4b9404e9a154fc99d7d71d1719813c3 2023-04-17 08:00:14,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c6cff3b039b64056b20612a1f4597f0b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c6cff3b039b64056b20612a1f4597f0b 2023-04-17 08:00:14,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/d87a01c5dc0942ebaabc1a989b590fa8 2023-04-17 08:00:14,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f99bc82c43bf4344813248257e6dbad6 2023-04-17 08:00:14,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/11dcecf9eba24bbdbb7343af9d4bdeb1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/11dcecf9eba24bbdbb7343af9d4bdeb1 2023-04-17 08:00:14,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e2fffa24e7d44e798d364a8f9308ba3 2023-04-17 08:00:14,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/41ac72008d6a48d68df611ade23e50ee 2023-04-17 08:00:14,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/26e74304e023480e99e72c4f61a66a5e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/26e74304e023480e99e72c4f61a66a5e 2023-04-17 08:00:14,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/4872471ed3234d6aafc7e8ce98d06d8e 2023-04-17 08:00:14,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5764439bc0564fdab56246b9765dbc0e 2023-04-17 08:00:14,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2f0ce7dfff814c40921d06c73aed3404 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2f0ce7dfff814c40921d06c73aed3404 2023-04-17 08:00:14,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef6064f0d09e4473b4125eb8f508c90d 2023-04-17 08:00:14,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/8ebc91a5197948eda10db4cd2f14a7bc 2023-04-17 08:00:14,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6f7a263d6c2540a79f5bfa0e993c9bfc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6f7a263d6c2540a79f5bfa0e993c9bfc 2023-04-17 08:00:14,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a41fd54e8cd740119136f646bbfcd342 2023-04-17 08:00:14,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/7db7f35b75ad475cae5606c72b2c2f33 2023-04-17 08:00:14,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/72e7ccc3b962440c87aef65210556807 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/72e7ccc3b962440c87aef65210556807 2023-04-17 08:00:14,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/2a365a53a8db493090c9399b6182c2e9 2023-04-17 08:00:14,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/eae08552615d481ea20c0de64ca0d3b9 2023-04-17 08:00:14,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e248e479f3a54e1c9319fa1c0c1b1281 2023-04-17 08:00:14,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e2ff0f140c684f9f883cf7311e3f80d9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/e2ff0f140c684f9f883cf7311e3f80d9 2023-04-17 08:00:14,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/a12038c613b74213a03c74340dbcd1ef 2023-04-17 08:00:14,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/c13ca4cea2c24d7e8c77eac2717748cf 2023-04-17 08:00:14,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f402f80c396347318653bb2b1bf3c338 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/f402f80c396347318653bb2b1bf3c338 2023-04-17 08:00:14,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/5e95cd2ecd064ac2b1b490b9c0097b59 2023-04-17 08:00:14,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/dbf03e2e7e994c2685982b2e89f56295 2023-04-17 08:00:14,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/086fb90b18844fe8849a2ff8ca8a3a11 2023-04-17 08:00:14,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/173c58cea9c64dc58f2782dc4d4530d8 2023-04-17 08:00:14,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/recovered.edits/1422.seqid, newMaxSeqId=1422, maxSeqId=1 2023-04-17 08:00:14,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a. 2023-04-17 08:00:14,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 2a1678219a4c91eb2f28c4b713537f8a: 2023-04-17 08:00:14,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:14,241 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2a1678219a4c91eb2f28c4b713537f8a, regionState=CLOSED 2023-04-17 08:00:14,241 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718414241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718414241"}]},"ts":"1681718414241"} 2023-04-17 08:00:14,243 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-17 08:00:14,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 2a1678219a4c91eb2f28c4b713537f8a, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.6740 sec 2023-04-17 08:00:14,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-17 08:00:14,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a1678219a4c91eb2f28c4b713537f8a, UNASSIGN in 1.6790 sec 2023-04-17 08:00:14,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718414245"}]},"ts":"1681718414245"} 2023-04-17 08:00:14,247 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:00:14,257 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:00:14,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7320 sec 2023-04-17 08:00:14,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 08:00:14,642 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-17 08:00:14,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:00:14,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,662 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,664 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 08:00:14,668 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:14,671 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/recovered.edits] 2023-04-17 08:00:14,673 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/013bc24c636a4b81872d8b05943a757b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/013bc24c636a4b81872d8b05943a757b 2023-04-17 08:00:14,674 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/f52edcc3f6ce4425b2022fd95faac305 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/A/f52edcc3f6ce4425b2022fd95faac305 2023-04-17 08:00:14,676 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/416b94195a694670a353ff90bc17e20f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/416b94195a694670a353ff90bc17e20f 2023-04-17 08:00:14,678 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7fd715dd5ed4b24bb59880fc6cadf3c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/B/d7fd715dd5ed4b24bb59880fc6cadf3c 2023-04-17 08:00:14,679 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/397c816d643a4ff38d8ab8fa969e2428 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/397c816d643a4ff38d8ab8fa969e2428 2023-04-17 08:00:14,680 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6ddf163732d847929f58d678f66ab61c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/6ddf163732d847929f58d678f66ab61c 2023-04-17 08:00:14,681 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b15267bec63f42cab5152cf994eb65ab to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b15267bec63f42cab5152cf994eb65ab 2023-04-17 08:00:14,682 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b313f079abde4d449ddbb534231cda79 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/b313f079abde4d449ddbb534231cda79 2023-04-17 08:00:14,683 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef0fa4b184c04e3aaabdc049b3401e64 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/C/ef0fa4b184c04e3aaabdc049b3401e64 2023-04-17 08:00:14,686 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/recovered.edits/1422.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a/recovered.edits/1422.seqid 2023-04-17 08:00:14,686 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/2a1678219a4c91eb2f28c4b713537f8a 2023-04-17 08:00:14,686 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:00:14,688 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,705 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:00:14,740 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:00:14,741 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,741 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:00:14,741 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718414741"}]},"ts":"9223372036854775807"} 2023-04-17 08:00:14,744 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:00:14,744 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2a1678219a4c91eb2f28c4b713537f8a, NAME => 'TestAcidGuarantees,,1681718388417.2a1678219a4c91eb2f28c4b713537f8a.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:00:14,744 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:00:14,744 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718414744"}]},"ts":"9223372036854775807"} 2023-04-17 08:00:14,746 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:00:14,758 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 106 msec 2023-04-17 08:00:14,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 08:00:14,766 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-17 08:00:14,787 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 207) Potentially hanging thread: (jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52b7c078-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1359960654_15 at /127.0.0.1:57026 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase16:39843-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1359960654_15 at /127.0.0.1:56986 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52b7c078-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1359960654_15 at /127.0.0.1:57126 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52b7c078-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52b7c078-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=582 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 338) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=8023 (was 8606) 2023-04-17 08:00:14,800 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=582, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=167, AvailableMemoryMB=8023 2023-04-17 08:00:14,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:00:14,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:00:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:14,806 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 08:00:14,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-17 08:00:14,807 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 08:00:14,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:14,808 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:14,809 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 empty. 2023-04-17 08:00:14,809 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:14,809 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:00:14,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:15,219 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:00:15,222 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f26affd8ee8c5b2a0826f19ce7699b72, NAME => 'TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 08:00:15,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:15,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:00:15,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f26affd8ee8c5b2a0826f19ce7699b72, disabling compactions & flushes 2023-04-17 08:00:15,634 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:15,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:15,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. after waiting 0 ms 2023-04-17 08:00:15,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:15,635 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:15,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:15,640 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 08:00:15,643 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718415643"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718415643"}]},"ts":"1681718415643"} 2023-04-17 08:00:15,646 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 08:00:15,647 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 08:00:15,647 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718415647"}]},"ts":"1681718415647"} 2023-04-17 08:00:15,648 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 08:00:15,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, ASSIGN}] 2023-04-17 08:00:15,694 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, ASSIGN 2023-04-17 08:00:15,695 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 08:00:15,847 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:15,847 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718415847"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718415847"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718415847"}]},"ts":"1681718415847"} 2023-04-17 08:00:15,852 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:15,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:16,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:16,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f26affd8ee8c5b2a0826f19ce7699b72, NAME => 'TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:00:16,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:00:16,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,012 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,014 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A 2023-04-17 08:00:16,014 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A 2023-04-17 08:00:16,014 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:16,015 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName A 2023-04-17 08:00:16,015 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:16,015 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,017 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B 2023-04-17 08:00:16,017 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B 2023-04-17 08:00:16,018 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:16,018 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName B 2023-04-17 08:00:16,019 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:16,019 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,020 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C 2023-04-17 08:00:16,020 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C 2023-04-17 08:00:16,021 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:16,021 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName C 2023-04-17 08:00:16,022 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:16,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:16,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:00:16,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:16,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 08:00:16,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened f26affd8ee8c5b2a0826f19ce7699b72; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60047102, jitterRate=-0.10522845387458801}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:00:16,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:16,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., pid=18, masterSystemTime=1681718416006 2023-04-17 08:00:16,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:16,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:16,030 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:16,030 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718416030"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718416030"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718416030"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718416030"}]},"ts":"1681718416030"} 2023-04-17 08:00:16,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-17 08:00:16,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 in 181 msec 2023-04-17 08:00:16,034 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-17 08:00:16,034 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, ASSIGN in 343 msec 2023-04-17 08:00:16,035 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 08:00:16,035 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718416035"}]},"ts":"1681718416035"} 2023-04-17 08:00:16,036 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 08:00:16,046 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 08:00:16,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2420 sec 2023-04-17 08:00:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 08:00:16,917 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-17 08:00:16,919 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x556231fa to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:16,957 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@671fd08e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:16,960 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:16,967 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:16,969 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 08:00:16,973 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 08:00:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:00:16,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:00:16,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:17,402 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:00:17,403 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:00:17,403 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:00:17,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 08:00:17,417 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, REOPEN/MOVE}] 2023-04-17 08:00:17,418 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, REOPEN/MOVE 2023-04-17 08:00:17,418 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:17,418 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718417418"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718417418"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718417418"}]},"ts":"1681718417418"} 2023-04-17 08:00:17,420 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:17,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing f26affd8ee8c5b2a0826f19ce7699b72, disabling compactions & flushes 2023-04-17 08:00:17,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. after waiting 0 ms 2023-04-17 08:00:17,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 08:00:17,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:17,579 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: f26affd8ee8c5b2a0826f19ce7699b72 to self. 2023-04-17 08:00:17,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,582 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=CLOSED 2023-04-17 08:00:17,583 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718417582"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718417582"}]},"ts":"1681718417582"} 2023-04-17 08:00:17,586 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-17 08:00:17,586 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 in 164 msec 2023-04-17 08:00:17,587 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=true 2023-04-17 08:00:17,738 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:17,739 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718417738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718417738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718417738"}]},"ts":"1681718417738"} 2023-04-17 08:00:17,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:17,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f26affd8ee8c5b2a0826f19ce7699b72, NAME => 'TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:00:17,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:00:17,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,907 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,909 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A 2023-04-17 08:00:17,909 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A 2023-04-17 08:00:17,909 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:17,912 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName A 2023-04-17 08:00:17,914 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:17,915 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,917 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B 2023-04-17 08:00:17,917 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B 2023-04-17 08:00:17,917 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:17,917 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName B 2023-04-17 08:00:17,918 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:17,918 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,919 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C 2023-04-17 08:00:17,919 DEBUG [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C 2023-04-17 08:00:17,919 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:17,919 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f26affd8ee8c5b2a0826f19ce7699b72 columnFamilyName C 2023-04-17 08:00:17,920 INFO [StoreOpener-f26affd8ee8c5b2a0826f19ce7699b72-1] regionserver.HStore(310): Store=f26affd8ee8c5b2a0826f19ce7699b72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:17,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:00:17,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:17,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened f26affd8ee8c5b2a0826f19ce7699b72; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62324414, jitterRate=-0.07129386067390442}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:17,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., pid=23, masterSystemTime=1681718417896 2023-04-17 08:00:17,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,928 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:17,928 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:17,929 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718417928"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718417928"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718417928"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718417928"}]},"ts":"1681718417928"} 2023-04-17 08:00:17,931 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-17 08:00:17,931 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 in 188 msec 2023-04-17 08:00:17,932 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-17 08:00:17,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, REOPEN/MOVE in 514 msec 2023-04-17 08:00:17,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-17 08:00:17,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 528 msec 2023-04-17 08:00:17,936 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 951 msec 2023-04-17 08:00:17,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-17 08:00:17,946 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x389356cd to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:17,979 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:17,980 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x595355a0 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,065 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7acc1387, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,066 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x38da9522 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,307 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7be739cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,309 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x0a84dba6 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,375 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@486fba63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,376 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x7c0b7748 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,392 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50feca84, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,393 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,408 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,408 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x52902392 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,422 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@651f745a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,422 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x4e1da5ba to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,432 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d429358, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,433 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x17a1bd04 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:18,446 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3432fa3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:18,456 DEBUG [hconnection-0x73b9576b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,456 DEBUG [hconnection-0x49850285-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,456 DEBUG [hconnection-0x6a4511ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,457 DEBUG [hconnection-0x3b216fe0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,457 DEBUG [hconnection-0x3d42b3fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,458 DEBUG [hconnection-0x316a72c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,458 DEBUG [hconnection-0x93aac27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,458 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55242, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,465 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,466 DEBUG [hconnection-0x5845f0ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,466 DEBUG [hconnection-0xfbc83da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:18,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,466 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,466 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:18,467 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,468 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,468 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:18,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:18,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:18,477 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:18,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:18,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:18,478 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,478 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:18,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:00:18,493 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,493 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:18,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:18,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:18,494 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:18,494 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,494 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:18,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:18,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:18,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:18,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:18,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:18,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107687bf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:18,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:18,509 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,515 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:18,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:18,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:18,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:18,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:18,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:18,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:18,526 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:18,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:18,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:18,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:18,526 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:18,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681718418504, max timestamp=1681718418526Num uniques 859; 2023-04-17 08:00:18,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:00:18,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681718418504, max timestamp=1681718418526Num uniques 859; 2023-04-17 08:00:18,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:00:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718418523, max timestamp=1681718418526Num uniques 859; 2023-04-17 08:00:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:18,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718478558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718478560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718478561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,567 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718478563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:18,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718478563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:18,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:18,567 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,568 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,569 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,569 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:18,569 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:18,669 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:18,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718478666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:18,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718478667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718478668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718478669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718478669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:18,680 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:18,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:18,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:18,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:18,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:18,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:18,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:18,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:18,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:18,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:18,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:18,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:18,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cde7bf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:18,709 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:18,709 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:18,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:18,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:18,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:18,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:18,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:18,726 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:18,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:18,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:18,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:18,726 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:18,731 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,731 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,732 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,734 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,745 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:18,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:18,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:18,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,746 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:18,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,746 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:18,846 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:18,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:18,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:18,868 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:18,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:18,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:18,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,868 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718478871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718478872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718478873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718478873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:18,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718478874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,878 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:18,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:18,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:18,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:18,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:18,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:18,892 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:18,892 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:18,892 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,892 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:18,892 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:18,892 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:18,892 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:18,892 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:18,892 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:18,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:18,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,906 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:18,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e8641c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:18,908 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:18,908 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,915 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:18,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:18,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:18,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:18,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:18,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:18,927 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:18,927 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:18,927 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:18,927 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:18,927 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:18,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:18,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:18,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:18,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:18,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:18,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:18,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:18,948 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:18,948 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,949 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:18,949 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:18,949 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:18,962 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c73181c0d9b84a9d94e3d20a55724d42 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c73181c0d9b84a9d94e3d20a55724d42 2023-04-17 08:00:19,049 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,056 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,065 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,066 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,066 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,066 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,066 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:19,066 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,067 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:19,067 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:19,067 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,067 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:19,067 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:19,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:19,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@441c2883[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:19,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:19,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:19,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:19,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:19,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:19,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:19,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:19,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:19,091 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:19,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:19,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:19,092 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:19,092 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:19,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:19,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:19,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,128 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:19,128 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718479175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718479175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718479175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718479176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718479177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,228 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,232 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,233 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,245 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:19,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:19,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:19,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:19,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,257 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,258 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,258 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:19,259 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:19,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f304c0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:19,259 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,265 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:19,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:19,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:19,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:19,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,274 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:19,274 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,274 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:19,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:19,276 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:19,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:19,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:19,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:19,276 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:19,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,284 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,295 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:19,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:19,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,296 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:19,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,296 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:19,371 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/17686e608318409e95e4f594bd5ba60b 2023-04-17 08:00:19,396 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,400 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,406 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,408 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,412 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:19,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:19,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:19,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:19,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:19,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,423 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,423 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,423 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:19,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96ae77e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:19,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:19,425 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,431 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:19,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:19,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:19,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:19,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:19,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:19,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:19,442 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:19,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:19,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:19,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:19,442 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:19,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:19,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:19,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:19,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,462 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,462 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,562 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,566 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,575 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,575 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,575 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:19,575 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:19,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:19,581 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb5aacb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:19,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:19,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,590 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:19,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:19,590 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:19,592 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,592 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:19,603 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:19,603 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,603 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:19,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:19,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:19,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:19,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:19,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:19,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:19,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,616 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:19,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,623 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,623 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:19,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,624 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,624 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,624 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718479679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718479681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718479682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718479684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718479685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,724 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,727 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,727 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,727 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,727 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,727 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,727 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,751 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:19,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:19,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:19,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,771 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:19,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc8c1ae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:19,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:19,775 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,781 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 08:00:19,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/d86368bc42a046a791cbdd9fc6c2c6f5 2023-04-17 08:00:19,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:19,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:19,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:19,785 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,785 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:19,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:19,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:19,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:19,798 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:19,798 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:19,798 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:19,798 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:19,798 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:19,806 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,806 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,808 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:19,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:19,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:19,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:19,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:19,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/3c30c3c1e852478687a65859e34117a4 2023-04-17 08:00:19,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:19,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:19,821 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:19,822 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:19,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:19,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/17686e608318409e95e4f594bd5ba60b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b 2023-04-17 08:00:19,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:19,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b, entries=150, sequenceid=15, filesize=25.1 K 2023-04-17 08:00:19,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/d86368bc42a046a791cbdd9fc6c2c6f5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5 2023-04-17 08:00:19,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 08:00:19,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/3c30c3c1e852478687a65859e34117a4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4 2023-04-17 08:00:19,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 08:00:19,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for f26affd8ee8c5b2a0826f19ce7699b72 in 1360ms, sequenceid=15, compaction requested=false 2023-04-17 08:00:19,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:19,922 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:19,926 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:19,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:19,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:19,927 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,927 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:19,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:19,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:19,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:19,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:19,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:19,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:19,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:19,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:19,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:19,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:19,944 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:19,944 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:19,944 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:19,944 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:20,377 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a222f172b0dd46b2b3b7f714121994d3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a222f172b0dd46b2b3b7f714121994d3 2023-04-17 08:00:20,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:20,686 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:20,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718480694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718480697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718480697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718480697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718480698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,785 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/4f1742ca91a546d1b8037a1f1cbade19 2023-04-17 08:00:20,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718480799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718480802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:20,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718480802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718480802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,827 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/fee8e47ab49e48689f0a926ca039fbae 2023-04-17 08:00:20,862 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/a13b4f41fcef4aacb4e0e046a1723a57 2023-04-17 08:00:20,869 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/4f1742ca91a546d1b8037a1f1cbade19 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19 2023-04-17 08:00:20,876 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19, entries=150, sequenceid=51, filesize=25.1 K 2023-04-17 08:00:20,878 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/fee8e47ab49e48689f0a926ca039fbae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae 2023-04-17 08:00:20,890 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae, entries=150, sequenceid=51, filesize=11.6 K 2023-04-17 08:00:20,891 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/a13b4f41fcef4aacb4e0e046a1723a57 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57 2023-04-17 08:00:20,900 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57, entries=150, sequenceid=51, filesize=11.6 K 2023-04-17 08:00:20,901 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=60.38 KB/61830 for f26affd8ee8c5b2a0826f19ce7699b72 in 957ms, sequenceid=51, compaction requested=false 2023-04-17 08:00:20,901 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:20,902 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:20,902 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:20,902 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:20,902 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:20,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:20,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:20,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:20,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:20,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e51713[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:20,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:20,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,924 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:20,924 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:20,924 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:20,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:20,932 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:20,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:20,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:20,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:20,935 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:20,935 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:20,935 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:20,935 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:20,936 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:20,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:20,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:20,946 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:20,950 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,956 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:20,956 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:20,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:20,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:20,957 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:20,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:20,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,957 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:20,957 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:20,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:20,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:21,007 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:21,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:21,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718481029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718481030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718481034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718481035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,057 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,062 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,073 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,075 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,075 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,075 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,075 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,075 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,082 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dea4906[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,090 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:21,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:21,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:21,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:21,101 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:21,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:21,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:21,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:21,101 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:21,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:21,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,124 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:21,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,124 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:21,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718481137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718481137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718481137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718481137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,224 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,227 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,227 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,227 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,227 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,238 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,238 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba4b82[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,251 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,275 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,275 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,275 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,275 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:21,275 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,276 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,287 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:21,287 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,287 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:21,289 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:21,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:21,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:21,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:21,289 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:21,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,300 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:21,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:21,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,307 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,307 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:21,307 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:21,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718481340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718481340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718481340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718481340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,407 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,410 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,411 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,411 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,422 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,427 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177d6dc575981c409b87c0bbd175b3b7d4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177d6dc575981c409b87c0bbd175b3b7d4 2023-04-17 08:00:21,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,431 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,432 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53023a4a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:21,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:21,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:21,451 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:21,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:21,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:21,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:21,451 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:21,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:21,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:21,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,474 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:21,474 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,474 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,574 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,576 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,577 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,577 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,577 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,577 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,588 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,588 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,588 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,588 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,598 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b4e2b6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,599 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:21,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:21,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:21,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:21,625 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:21,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:21,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:21,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:21,625 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:21,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,632 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,640 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:21,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,640 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,640 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:21,640 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718481642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718481643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718481645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718481646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,740 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,743 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,743 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,743 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,743 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,744 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,744 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,753 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,754 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,754 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,756 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,756 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,756 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,756 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,765 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d92edf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,773 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,774 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,774 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:21,818 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,818 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,818 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,818 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:21,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:21,821 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:21,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:21,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:21,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:21,821 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:21,838 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/d0c73298996e4d319db1ea0992b3bc8b 2023-04-17 08:00:21,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,852 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,852 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,857 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:21,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:21,865 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:21,865 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:21,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/0a62cc5b22ad4475aaa728db81008722 2023-04-17 08:00:21,965 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:21,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:21,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:21,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:21,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:21,968 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:21,968 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,968 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:21,968 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:21,969 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,969 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:21,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:21,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:21,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:21,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:21,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:21,980 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,980 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:21,980 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:21,980 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:21,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:21,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:21,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:21,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:21,990 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:21,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:21,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:21,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc39dfd[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:21,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:21,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:21,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:21,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:21,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:21,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:21,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:22,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:22,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,006 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:22,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:22,008 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:22,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:22,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:22,009 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:22,009 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:22,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:22,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,023 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,023 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,124 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:22,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:22,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:22,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:22,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:22,127 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:22,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:22,127 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:22,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:22,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:22,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:22,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:22,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:22,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:22,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:22,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:22,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:22,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:22,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:22,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:22,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:22,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:22,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718482143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:22,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495136a5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:22,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:22,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:22,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718482148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:22,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718482151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:22,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718482151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,156 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:22,156 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:22,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:22,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:22,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:22,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:22,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:22,167 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:22,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:22,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:22,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:22,167 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:22,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,181 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,181 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:22,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:22,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:22,181 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,182 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:22,182 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:22,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,182 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,282 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:22,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:22,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:22,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:22,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:22,285 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:22,285 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,285 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:22,285 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:22,285 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,285 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/15c8f871beba4c7a8762101bc7478330 2023-04-17 08:00:22,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/d0c73298996e4d319db1ea0992b3bc8b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b 2023-04-17 08:00:22,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,295 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b, entries=200, sequenceid=64, filesize=31.9 K 2023-04-17 08:00:22,295 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,295 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/0a62cc5b22ad4475aaa728db81008722 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722 2023-04-17 08:00:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:22,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:22,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:22,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:22,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:22,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:22,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:22,298 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,298 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:22,298 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:22,299 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:22,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:22,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:22,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:22,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722, entries=150, sequenceid=64, filesize=11.6 K 2023-04-17 08:00:22,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/15c8f871beba4c7a8762101bc7478330 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330 2023-04-17 08:00:22,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:22,309 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330, entries=150, sequenceid=64, filesize=11.6 K 2023-04-17 08:00:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:22,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56efdc6b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:22,311 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:22,311 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f26affd8ee8c5b2a0826f19ce7699b72 in 1304ms, sequenceid=64, compaction requested=true 2023-04-17 08:00:22,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:22,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:22,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:22,311 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:22,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:22,311 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:22,313 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:22,313 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:22,313 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:22,313 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:22,313 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,313 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,313 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=82.2 K 2023-04-17 08:00:22,313 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=34.8 K 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 17686e608318409e95e4f594bd5ba60b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681718418485 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d86368bc42a046a791cbdd9fc6c2c6f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681718418485 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4f1742ca91a546d1b8037a1f1cbade19, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681718418534 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fee8e47ab49e48689f0a926ca039fbae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681718418534 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d0c73298996e4d319db1ea0992b3bc8b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:22,314 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0a62cc5b22ad4475aaa728db81008722, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:22,320 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,321 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:22,321 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:22,321 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:22,321 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,321 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:22,325 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#125 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:22,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:22,331 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:22,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:22,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:22,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:22,331 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:22,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,340 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,357 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#126 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:22,362 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/c67223440fb1482aa95633dc3890a980 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/c67223440fb1482aa95633dc3890a980 2023-04-17 08:00:22,368 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,368 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:22,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:22,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:22,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,369 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:22,369 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into c67223440fb1482aa95633dc3890a980(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:22,369 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:22,369 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=13, startTime=1681718422311; duration=0sec 2023-04-17 08:00:22,370 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:22,370 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:22,369 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:22,371 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:22,371 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:22,371 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,371 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=34.8 K 2023-04-17 08:00:22,372 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3c30c3c1e852478687a65859e34117a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681718418485 2023-04-17 08:00:22,372 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a13b4f41fcef4aacb4e0e046a1723a57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681718418534 2023-04-17 08:00:22,373 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 15c8f871beba4c7a8762101bc7478330, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:22,383 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#127 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:22,404 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/99cc1d8129f4448d850f4f5893a3b1b2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/99cc1d8129f4448d850f4f5893a3b1b2 2023-04-17 08:00:22,409 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into 99cc1d8129f4448d850f4f5893a3b1b2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:22,409 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:22,409 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=13, startTime=1681718422311; duration=0sec 2023-04-17 08:00:22,409 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:22,470 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:22,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:22,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:22,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:22,473 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:22,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:22,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:22,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:22,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:22,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:22,483 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:22,483 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,483 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:22,483 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:22,483 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:22,483 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,483 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:22,484 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:22,484 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,508 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e66c6649c9c9405cb7958fd7d29ca459 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e66c6649c9c9405cb7958fd7d29ca459 2023-04-17 08:00:22,517 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6a964c0a171549f1b071681432f5d8a1 2023-04-17 08:00:22,541 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/f3785a9ca9914cac8179d24536d306a0 2023-04-17 08:00:22,557 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/44552e23d0f8447fabd7ad6543a5de1d 2023-04-17 08:00:22,562 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6a964c0a171549f1b071681432f5d8a1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1 2023-04-17 08:00:22,567 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1, entries=150, sequenceid=85, filesize=25.1 K 2023-04-17 08:00:22,568 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/f3785a9ca9914cac8179d24536d306a0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0 2023-04-17 08:00:22,589 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0, entries=150, sequenceid=85, filesize=11.6 K 2023-04-17 08:00:22,593 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/44552e23d0f8447fabd7ad6543a5de1d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d 2023-04-17 08:00:22,606 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d, entries=150, sequenceid=85, filesize=11.6 K 2023-04-17 08:00:22,607 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for f26affd8ee8c5b2a0826f19ce7699b72 in 124ms, sequenceid=85, compaction requested=false 2023-04-17 08:00:22,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:22,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:22,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:22,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:22,608 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:22,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:22,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68febfbf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:22,617 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:22,617 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:22,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:22,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:22,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:22,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,632 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:22,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:22,634 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:22,634 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:22,634 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:22,634 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:22,634 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:22,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:22,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:22,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,657 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:22,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,657 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,657 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,757 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:22,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:22,761 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:22,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:22,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:22,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,770 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:22,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:22,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:22,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:22,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:22,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:22,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:22,784 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:22,784 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,784 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:22,784 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:22,785 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,821 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417107df9f6a25e4b8eb7866336f0648071 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417107df9f6a25e4b8eb7866336f0648071 2023-04-17 08:00:22,827 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=90, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/da465b55a5c14f969a6865a908133d06 2023-04-17 08:00:22,850 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/8a60fcb6cc2545c9a22a0aea16c17727 2023-04-17 08:00:22,868 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/061e48e6773b473fbf911dae2f890aa2 2023-04-17 08:00:22,875 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/da465b55a5c14f969a6865a908133d06 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06 2023-04-17 08:00:22,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:22,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:22,880 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06, entries=100, sequenceid=90, filesize=18.4 K 2023-04-17 08:00:22,882 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/8a60fcb6cc2545c9a22a0aea16c17727 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727 2023-04-17 08:00:22,889 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727, entries=50, sequenceid=90, filesize=7.0 K 2023-04-17 08:00:22,890 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/061e48e6773b473fbf911dae2f890aa2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2 2023-04-17 08:00:22,894 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2, entries=50, sequenceid=90, filesize=7.0 K 2023-04-17 08:00:22,895 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=53.67 KB/54960 for f26affd8ee8c5b2a0826f19ce7699b72 in 111ms, sequenceid=90, compaction requested=true 2023-04-17 08:00:22,895 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:22,895 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:22,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:22,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:22,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:22,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:22,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:22,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:22,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:22,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:22,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f45eb28[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:22,906 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:22,906 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,915 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:22,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:22,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:22,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:22,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:22,925 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:22,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:22,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:22,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:22,925 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:22,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:22,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:22,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:22,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:22,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:22,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:22,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:22,941 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:22,941 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:22,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718482965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,041 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:23,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:23,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:23,044 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:23,044 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,044 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:23,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:23,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,053 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,053 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:23,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:23,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:23,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:23,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:23,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:23,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:23,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:23,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe10e7b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:23,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718483069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:23,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:23,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:23,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:23,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:23,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:23,114 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:23,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:23,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:23,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:23,114 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:23,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:23,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,148 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:23,148 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:23,149 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,150 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:23,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718483153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718483154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718483159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718483164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,232 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/3a327a2b060a42c0a1f21d27c7111614 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3a327a2b060a42c0a1f21d27c7111614 2023-04-17 08:00:23,238 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into 3a327a2b060a42c0a1f21d27c7111614(size=25.2 K), total size for store is 68.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:23,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:23,238 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=13, startTime=1681718422311; duration=0sec 2023-04-17 08:00:23,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:23,249 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:23,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:23,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:23,262 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:23,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:23,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:23,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:23,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:23,275 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:23,276 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,276 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:23,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718483276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:23,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:23,288 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,288 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:23,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:23,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:23,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:23,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:23,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:23,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,300 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,300 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:23,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:23,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17459168[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:23,300 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:23,300 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:23,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:23,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:23,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:23,315 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,316 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417cb16edd824764e319809f1476fe6c887 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cb16edd824764e319809f1476fe6c887 2023-04-17 08:00:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:23,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:23,318 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:23,318 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:23,318 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:23,318 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:23,318 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:23,335 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/3fbd8aaf16224c879a927b9d70e91f84 2023-04-17 08:00:23,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/46aebfd4101340129c2fefa7ec530044 2023-04-17 08:00:23,353 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,353 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,354 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,358 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:23,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:23,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,387 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,387 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:23,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,487 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:23,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:23,490 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:23,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:23,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:23,491 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,491 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:23,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:23,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:23,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:23,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:23,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:23,499 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:23,499 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,499 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:23,500 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:23,500 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:23,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:23,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:23,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:23,506 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,506 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:23,508 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:23,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@adbd778[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:23,508 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,514 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:23,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:23,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:23,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:23,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:23,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:23,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:23,525 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:23,525 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:23,525 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:23,525 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:23,525 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:23,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,535 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,545 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,545 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:23,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,545 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:23,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718483584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,646 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:23,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:23,648 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:23,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:23,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:23,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:23,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:23,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:23,663 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:23,663 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,663 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:23,663 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:23,663 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:23,663 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,664 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:23,664 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:23,664 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:23,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:23,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:23,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:23,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:23,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a1771a0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:23,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:23,675 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,681 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:23,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:23,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:23,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:23,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:23,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:23,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:23,691 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:23,691 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:23,691 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:23,691 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:23,691 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:23,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:23,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:23,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,707 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:23,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:23,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:23,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:23,707 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:23,707 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/1f10e0fcff4b4f87bd73621ca22287ae 2023-04-17 08:00:23,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/3fbd8aaf16224c879a927b9d70e91f84 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84 2023-04-17 08:00:23,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84, entries=250, sequenceid=102, filesize=38.7 K 2023-04-17 08:00:23,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/46aebfd4101340129c2fefa7ec530044 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044 2023-04-17 08:00:23,807 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:23,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:23,817 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:23,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:23,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:23,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044, entries=150, sequenceid=102, filesize=11.6 K 2023-04-17 08:00:23,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/1f10e0fcff4b4f87bd73621ca22287ae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae 2023-04-17 08:00:23,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae, entries=150, sequenceid=102, filesize=11.6 K 2023-04-17 08:00:23,827 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f26affd8ee8c5b2a0826f19ce7699b72 in 929ms, sequenceid=102, compaction requested=true 2023-04-17 08:00:23,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:23,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:23,828 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:23,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:23,828 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:23,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:23,831 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110021 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:23,831 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:23,832 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,832 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3a327a2b060a42c0a1f21d27c7111614, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=107.4 K 2023-04-17 08:00:23,832 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42961 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:23,832 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:23,832 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,832 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/c67223440fb1482aa95633dc3890a980, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=42.0 K 2023-04-17 08:00:23,832 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3a327a2b060a42c0a1f21d27c7111614, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting c67223440fb1482aa95633dc3890a980, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6a964c0a171549f1b071681432f5d8a1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718421028 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f3785a9ca9914cac8179d24536d306a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718421028 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting da465b55a5c14f969a6865a908133d06, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681718422751 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 8a60fcb6cc2545c9a22a0aea16c17727, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681718422783 2023-04-17 08:00:23,833 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3fbd8aaf16224c879a927b9d70e91f84, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422855 2023-04-17 08:00:23,834 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 46aebfd4101340129c2fefa7ec530044, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422864 2023-04-17 08:00:23,856 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,856 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:23,857 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#138 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,857 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:23,857 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:23,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:23,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:23,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:23,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:23,859 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:23,859 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:23,859 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:23,859 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:23,859 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:23,859 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,859 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:23,859 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:23,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:23,861 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#137 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:23,871 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/4ace0f5dcb314a9aa22881a23096de9f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/4ace0f5dcb314a9aa22881a23096de9f 2023-04-17 08:00:23,874 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d918965fc5b54e7486e670598846dfef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d918965fc5b54e7486e670598846dfef 2023-04-17 08:00:23,882 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into 4ace0f5dcb314a9aa22881a23096de9f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:23,882 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:23,883 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=12, startTime=1681718423828; duration=0sec 2023-04-17 08:00:23,883 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:23,883 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:00:23,886 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42961 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:00:23,897 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=123, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/5a4008b98e514331a0d50ef21efc3c14 2023-04-17 08:00:23,899 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:23,900 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:23,900 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/99cc1d8129f4448d850f4f5893a3b1b2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=42.0 K 2023-04-17 08:00:23,900 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 99cc1d8129f4448d850f4f5893a3b1b2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681718420693 2023-04-17 08:00:23,901 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 44552e23d0f8447fabd7ad6543a5de1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718421028 2023-04-17 08:00:23,901 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 061e48e6773b473fbf911dae2f890aa2, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681718422783 2023-04-17 08:00:23,902 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1f10e0fcff4b4f87bd73621ca22287ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422864 2023-04-17 08:00:23,917 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:23,939 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/b1f3dcca28b14c39a0b832768f27363e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b1f3dcca28b14c39a0b832768f27363e 2023-04-17 08:00:23,944 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into b1f3dcca28b14c39a0b832768f27363e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:23,945 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:23,945 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=12, startTime=1681718423828; duration=0sec 2023-04-17 08:00:23,945 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:24,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:24,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718484107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718484211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,285 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/017c7d892b3241c3824bc8dbf4c6c88c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/017c7d892b3241c3824bc8dbf4c6c88c 2023-04-17 08:00:24,295 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into 017c7d892b3241c3824bc8dbf4c6c88c(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:24,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:24,295 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=12, startTime=1681718423828; duration=0sec 2023-04-17 08:00:24,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:24,327 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/ce095c5bbbe74afe8f75abb832eb4f02 2023-04-17 08:00:24,345 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/10ff008de8624839b4a6e9845c705e62 2023-04-17 08:00:24,352 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/5a4008b98e514331a0d50ef21efc3c14 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14 2023-04-17 08:00:24,357 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14, entries=150, sequenceid=123, filesize=25.1 K 2023-04-17 08:00:24,358 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/ce095c5bbbe74afe8f75abb832eb4f02 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02 2023-04-17 08:00:24,365 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02, entries=150, sequenceid=123, filesize=11.6 K 2023-04-17 08:00:24,366 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/10ff008de8624839b4a6e9845c705e62 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62 2023-04-17 08:00:24,384 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62, entries=150, sequenceid=123, filesize=11.6 K 2023-04-17 08:00:24,385 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f26affd8ee8c5b2a0826f19ce7699b72 in 526ms, sequenceid=123, compaction requested=false 2023-04-17 08:00:24,385 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:24,386 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:24,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:24,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:24,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:24,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:00:24,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:24,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:24,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:24,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:24,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:24,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:24,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d323e4f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:24,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:24,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:24,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:24,440 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:24,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:24,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:24,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:24,448 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:24,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:24,453 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:24,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:24,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:24,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:24,453 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:24,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,465 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,466 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,469 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,469 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171f550c4387b6442ab4db0a78cbd44247 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f550c4387b6442ab4db0a78cbd44247 2023-04-17 08:00:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,482 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:24,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:24,483 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:24,483 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:24,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:24,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:24,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718484485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,583 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:24,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:24,586 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:24,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:24,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:24,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:24,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718484588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,595 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,595 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,595 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:24,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:24,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:24,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:24,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:24,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:24,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:24,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:24,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:24,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:24,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:24,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:24,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:24,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:24,606 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:24,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:24,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56c73c50[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:24,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,615 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:24,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:24,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:24,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:24,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:24,623 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:24,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:24,625 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:24,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:24,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:24,625 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:24,625 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:24,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:24,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,645 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:24,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,646 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:24,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,646 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,746 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:24,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:24,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:24,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:24,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:24,752 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:24,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:24,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:24,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,753 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,762 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:24,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:24,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:24,767 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:24,767 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,767 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:24,768 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,768 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:24,768 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:24,768 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:24,768 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:24,778 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,778 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:24,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f351e7[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:24,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:24,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,790 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:24,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:24,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:24,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:24,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:24,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718484791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,798 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:24,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:24,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:24,800 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:24,800 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:24,800 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:24,800 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:24,800 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:24,806 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,806 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,806 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:24,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,815 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:24,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,815 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,815 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,892 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b77e139c1599452a89f19bda8273c0d0 2023-04-17 08:00:24,915 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:24,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:24,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:24,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:24,918 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:24,918 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,918 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:24,918 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:24,919 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,919 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,928 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:24,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:24,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:24,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:24,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:24,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:24,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:24,957 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:24,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6388aa68[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:24,959 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:24,959 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:24,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:24,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:24,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:24,980 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:24,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:24,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:24,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:24,980 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:24,990 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,990 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:24,990 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:24,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:24,998 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:24,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:24,998 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:24,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:24,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:24,999 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718485092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,099 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:25,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:25,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:25,101 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:25,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:25,101 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:25,102 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,102 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:25,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:25,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:25,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:25,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:25,113 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,113 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:25,113 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:25,113 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:25,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:25,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:25,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:25,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f342010[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:25,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:25,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,131 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:25,131 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:25,132 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:25,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:25,140 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:25,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:25,143 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:25,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:25,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:25,143 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:25,143 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:25,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:25,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718485157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718485158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,161 DEBUG [Thread-910] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:25,164 DEBUG [Thread-916] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:25,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:25,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:25,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:25,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:25,166 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:25,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:25,166 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:25,166 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:25,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:25,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718485175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,178 DEBUG [Thread-914] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718485178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,182 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:25,266 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:25,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:25,269 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:25,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:25,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:25,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:25,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:25,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:25,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:25,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:25,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:25,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:25,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:25,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:25,290 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:25,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70feba1a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:25,291 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:25,291 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,298 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:25,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:25,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:25,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:25,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/5c541f96c7264e6ab25d28bf63b0cc38 2023-04-17 08:00:25,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:25,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:25,309 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:25,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:25,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:25,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:25,309 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:25,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/14865b668d5448f3bccac4124d8324e6 2023-04-17 08:00:25,315 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,315 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,315 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:25,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:25,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,316 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:25,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b77e139c1599452a89f19bda8273c0d0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0 2023-04-17 08:00:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,317 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0, entries=150, sequenceid=139, filesize=25.3 K 2023-04-17 08:00:25,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/5c541f96c7264e6ab25d28bf63b0cc38 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38 2023-04-17 08:00:25,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38, entries=150, sequenceid=139, filesize=11.8 K 2023-04-17 08:00:25,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/14865b668d5448f3bccac4124d8324e6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:25,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:25,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:25,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:25,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:25,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:25,329 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,329 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:25,329 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:25,329 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:25,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:25,329 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:25,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6, entries=150, sequenceid=139, filesize=11.8 K 2023-04-17 08:00:25,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f26affd8ee8c5b2a0826f19ce7699b72 in 915ms, sequenceid=139, compaction requested=true 2023-04-17 08:00:25,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:25,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:25,331 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:25,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:25,331 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:25,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:25,333 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:25,333 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:25,333 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,333 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/4ace0f5dcb314a9aa22881a23096de9f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=35.2 K 2023-04-17 08:00:25,334 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77628 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:25,334 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4ace0f5dcb314a9aa22881a23096de9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422864 2023-04-17 08:00:25,334 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:25,334 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,334 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/017c7d892b3241c3824bc8dbf4c6c88c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=75.8 K 2023-04-17 08:00:25,334 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ce095c5bbbe74afe8f75abb832eb4f02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1681718422945 2023-04-17 08:00:25,335 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5c541f96c7264e6ab25d28bf63b0cc38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:25,335 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 017c7d892b3241c3824bc8dbf4c6c88c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422864 2023-04-17 08:00:25,335 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5a4008b98e514331a0d50ef21efc3c14, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1681718422945 2023-04-17 08:00:25,336 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b77e139c1599452a89f19bda8273c0d0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:25,340 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#146 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:25,353 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#147 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:25,429 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:25,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:25,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:25,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:25,432 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:25,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:25,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:25,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,440 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:25,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:25,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:25,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:25,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:25,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:25,441 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:25,441 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:25,441 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:25,441 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:25,441 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:25,441 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:25,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:25,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:25,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:25,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718485620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718485723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:25,750 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/1e543368df194ccda99e5f8f3a4b0bcf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1e543368df194ccda99e5f8f3a4b0bcf 2023-04-17 08:00:25,755 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into 1e543368df194ccda99e5f8f3a4b0bcf(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:25,755 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:25,755 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=13, startTime=1681718425331; duration=0sec 2023-04-17 08:00:25,755 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:25,755 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:25,756 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:25,756 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:25,756 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:25,756 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b1f3dcca28b14c39a0b832768f27363e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=35.2 K 2023-04-17 08:00:25,757 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b1f3dcca28b14c39a0b832768f27363e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681718422864 2023-04-17 08:00:25,757 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 10ff008de8624839b4a6e9845c705e62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1681718422945 2023-04-17 08:00:25,757 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 14865b668d5448f3bccac4124d8324e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:25,763 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#149 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:25,773 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/404fa4426cbd4f9ca5f12b2a3a150a20 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/404fa4426cbd4f9ca5f12b2a3a150a20 2023-04-17 08:00:25,778 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into 404fa4426cbd4f9ca5f12b2a3a150a20(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:25,778 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/bf911c7cf81b47e285c1386333208bb5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/bf911c7cf81b47e285c1386333208bb5 2023-04-17 08:00:25,778 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:25,778 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=13, startTime=1681718425331; duration=0sec 2023-04-17 08:00:25,779 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:25,783 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into bf911c7cf81b47e285c1386333208bb5(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:25,783 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:25,783 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=13, startTime=1681718425330; duration=0sec 2023-04-17 08:00:25,783 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:25,863 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417534ed0c9134f4a23938cc10dbc815e4a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417534ed0c9134f4a23938cc10dbc815e4a 2023-04-17 08:00:25,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:25,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718485928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:26,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:26,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718486231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:26,281 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/ee3eece6ce8f400d973336268c77c59b 2023-04-17 08:00:26,693 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/6151830fdb024b039e0a62efbf88fd0a 2023-04-17 08:00:26,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:26,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55280 deadline: 1681718486733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,104 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/09beb80d10d64e7c9ad04d9bc1c56493 2023-04-17 08:00:27,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/ee3eece6ce8f400d973336268c77c59b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b 2023-04-17 08:00:27,112 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b, entries=150, sequenceid=158, filesize=25.3 K 2023-04-17 08:00:27,113 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/6151830fdb024b039e0a62efbf88fd0a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a 2023-04-17 08:00:27,117 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a, entries=150, sequenceid=158, filesize=11.8 K 2023-04-17 08:00:27,118 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/09beb80d10d64e7c9ad04d9bc1c56493 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493 2023-04-17 08:00:27,122 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493, entries=150, sequenceid=158, filesize=11.8 K 2023-04-17 08:00:27,123 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f26affd8ee8c5b2a0826f19ce7699b72 in 1682ms, sequenceid=158, compaction requested=false 2023-04-17 08:00:27,123 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:27,123 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:27,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:27,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:27,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,124 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:27,124 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:27,156 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:27,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,157 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:27,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:27,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689ab18b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:27,158 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:27,158 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:27,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:27,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:27,165 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:27,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:27,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:27,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:27,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:27,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:27,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:27,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:27,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:27,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:27,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:27,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:27,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:27,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:27,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:27,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:27,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,223 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:27,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:27,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:27,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:27,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:27,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:27,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:27,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:27,224 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,224 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:27,224 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:27,324 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:27,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:27,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:27,328 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:27,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:27,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:27,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:27,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:27,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:27,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:27,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:27,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:27,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:27,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:27,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:27,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:27,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:27,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:27,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:27,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:27,338 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:27,338 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:27,338 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:00:27,338 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:27,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:27,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:27,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:27,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:27,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:27,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:27,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:27,751 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171ce838096afb4972a6aee262c31c5abe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171ce838096afb4972a6aee262c31c5abe 2023-04-17 08:00:27,757 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/d3f084ad6de746ff85cf75109c23b0c4 2023-04-17 08:00:27,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:27,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718487770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:27,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:27,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718487874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:28,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718488077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,183 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/645bcc51f0f34d9686bedb23d09e654c 2023-04-17 08:00:28,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:28,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718488380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,602 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/98b336de1bde4083b06af36d02422de4 2023-04-17 08:00:28,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/d3f084ad6de746ff85cf75109c23b0c4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4 2023-04-17 08:00:28,610 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4, entries=150, sequenceid=174, filesize=25.3 K 2023-04-17 08:00:28,610 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/645bcc51f0f34d9686bedb23d09e654c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c 2023-04-17 08:00:28,614 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c, entries=150, sequenceid=174, filesize=11.8 K 2023-04-17 08:00:28,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/98b336de1bde4083b06af36d02422de4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4 2023-04-17 08:00:28,618 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4, entries=150, sequenceid=174, filesize=11.8 K 2023-04-17 08:00:28,619 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f26affd8ee8c5b2a0826f19ce7699b72 in 1281ms, sequenceid=174, compaction requested=true 2023-04-17 08:00:28,619 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:28,619 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:28,619 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:28,619 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:28,619 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:28,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:28,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,631 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:28,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:28,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:28,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e679d1a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:28,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:28,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,639 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:28,640 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,640 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:28,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:28,640 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:28,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:28,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:28,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:28,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:28,650 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:28,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:28,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:28,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:28,650 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:28,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,656 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:28,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:28,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:28,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:28,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:28,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:28,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:28,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:28,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:28,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:28,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,665 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:28,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:28,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:28,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:28,665 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:28,665 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:28,765 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:28,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:28,768 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:28,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:28,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:28,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:28,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:28,778 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:28,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:28,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:28,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:28,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:28,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:28,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:28,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:28,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:28,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:28,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:28,780 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:28,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:28,781 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:28,781 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:28,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:28,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:28,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:28,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718488907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718489009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718489183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,186 DEBUG [Thread-910] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:29,192 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178178d046e5e84b639f07cc743f2263ee to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178178d046e5e84b639f07cc743f2263ee 2023-04-17 08:00:29,197 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6f488f27d6624cf3959c6d811515f84f 2023-04-17 08:00:29,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718489195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718489198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,200 DEBUG [Thread-916] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:29,200 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:29,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718489212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718489214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,217 DEBUG [Thread-914] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:29,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:29,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718489517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,618 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/d144665de47e482e9ddb44014cd4e750 2023-04-17 08:00:29,641 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/cc7e80e7779c4351b5ef4e0e364c39e6 2023-04-17 08:00:29,646 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6f488f27d6624cf3959c6d811515f84f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f 2023-04-17 08:00:29,650 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f, entries=150, sequenceid=193, filesize=25.3 K 2023-04-17 08:00:29,651 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/d144665de47e482e9ddb44014cd4e750 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750 2023-04-17 08:00:29,655 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 08:00:29,656 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/cc7e80e7779c4351b5ef4e0e364c39e6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6 2023-04-17 08:00:29,660 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 08:00:29,660 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f26affd8ee8c5b2a0826f19ce7699b72 in 880ms, sequenceid=193, compaction requested=true 2023-04-17 08:00:29,660 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:29,660 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:29,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:29,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ed1066[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:29,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:29,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:29,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:29,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:29,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:29,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:29,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:29,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:29,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:29,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:29,717 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:29,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:29,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:29,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:29,717 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:29,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:29,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:29,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:29,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:29,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:29,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,740 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:29,740 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:29,840 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:29,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:29,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:29,843 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:29,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:29,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:29,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:29,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:29,853 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:29,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:29,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:29,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:29,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:29,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:29,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:29,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:29,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:29,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:29,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:29,855 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:29,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:29,874 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d22391ac68c845fe969c2c93fde4d002 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d22391ac68c845fe969c2c93fde4d002 2023-04-17 08:00:29,888 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/da633a9c895a4c65a1bad6b53694c2b4 2023-04-17 08:00:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:30,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:30,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:30,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718490063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:30,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718490164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,300 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/1288690b842248b2ac394c2c8b19f88d 2023-04-17 08:00:30,310 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/2a7037488c9d49e5a84264bab780ccd5 2023-04-17 08:00:30,314 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/da633a9c895a4c65a1bad6b53694c2b4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4 2023-04-17 08:00:30,318 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4, entries=150, sequenceid=206, filesize=25.3 K 2023-04-17 08:00:30,319 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/1288690b842248b2ac394c2c8b19f88d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d 2023-04-17 08:00:30,323 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d, entries=150, sequenceid=206, filesize=11.8 K 2023-04-17 08:00:30,324 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/2a7037488c9d49e5a84264bab780ccd5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5 2023-04-17 08:00:30,327 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5, entries=150, sequenceid=206, filesize=11.8 K 2023-04-17 08:00:30,328 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f26affd8ee8c5b2a0826f19ce7699b72 in 473ms, sequenceid=206, compaction requested=true 2023-04-17 08:00:30,328 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:30,328 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:30,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:30,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:30,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:30,364 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,364 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:30,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c65a0b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:30,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:30,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:30,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:30,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:30,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:30,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:30,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:30,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:30,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:30,381 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:30,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:30,383 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:30,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:30,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:30,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:30,383 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:30,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:30,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718490397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,409 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,409 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,410 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,411 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:30,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:30,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,423 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,423 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:30,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718490501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,523 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:30,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:30,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:30,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:30,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:30,526 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:30,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:30,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:30,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:30,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:30,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:30,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:30,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:30,533 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,533 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:30,533 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:30,533 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:30,533 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:30,539 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:30,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48edabd1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:30,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:30,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:30,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:30,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:30,548 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:30,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:30,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:30,558 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:30,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:30,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:30,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:30,558 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:30,564 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,564 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,564 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:30,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:30,573 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:30,573 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:30,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,573 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,673 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:30,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:30,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:30,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:30,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:30,676 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:30,676 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,676 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:30,676 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:30,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,677 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,686 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:30,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:30,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:30,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:30,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:30,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:30,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:30,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a90609b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:30,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:30,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:30,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718490704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:30,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:30,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:30,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:30,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:30,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:30,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:30,716 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:30,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:30,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:30,717 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:30,717 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:30,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:30,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:30,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:30,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,740 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:30,740 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:30,786 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178291736a0bfd4c08a7ccfd8186b68c92 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178291736a0bfd4c08a7ccfd8186b68c92 2023-04-17 08:00:30,792 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=226, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/766223a5285945ee805001459c4c23b4 2023-04-17 08:00:30,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/da0704e578314e6a8d7fc0777214d2c1 2023-04-17 08:00:30,840 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:30,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:30,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:30,843 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:30,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:30,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:30,844 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,844 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,853 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:30,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:30,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:30,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:30,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:30,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:30,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:30,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:30,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:30,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:30,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:30,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:30,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:30,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:30,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:30,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:30,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:30,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e6398f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:30,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:30,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:30,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:30,873 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:30,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:30,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:30,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:30,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:30,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:30,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:30,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:30,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:30,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:30,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,918 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,918 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:30,918 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:30,919 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:30,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:30,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:30,919 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:30,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:30,919 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:30,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:31,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718491008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,019 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:31,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:31,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:31,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:31,022 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:31,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:31,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:31,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,031 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:31,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:31,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:31,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:31,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,032 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:31,033 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,033 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:31,033 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:31,033 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:31,033 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:31,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:31,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65b739da[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:31,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:31,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:31,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:31,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:31,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:31,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:31,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:31,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:31,058 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:31,058 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:31,058 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:31,058 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:31,058 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:31,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,069 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,078 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:31,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:31,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,079 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:31,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,079 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:31,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,179 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:31,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:31,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:31,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:31,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:31,182 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:31,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:31,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:31,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:31,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:31,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:31,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:31,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:31,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:31,192 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,192 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:31,192 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:31,192 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:31,192 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:31,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4021b59e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:31,203 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:31,203 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,211 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:31,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:31,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:31,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:31,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/6c2401bf114b4e3abd220e9e018f0373 2023-04-17 08:00:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:31,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:31,223 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:31,223 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:31,223 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:31,223 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:31,223 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:31,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/766223a5285945ee805001459c4c23b4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4 2023-04-17 08:00:31,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4, entries=200, sequenceid=226, filesize=32.1 K 2023-04-17 08:00:31,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/da0704e578314e6a8d7fc0777214d2c1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1 2023-04-17 08:00:31,231 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,231 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,232 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,234 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1, entries=150, sequenceid=226, filesize=11.8 K 2023-04-17 08:00:31,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/6c2401bf114b4e3abd220e9e018f0373 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373 2023-04-17 08:00:31,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373, entries=150, sequenceid=226, filesize=11.8 K 2023-04-17 08:00:31,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f26affd8ee8c5b2a0826f19ce7699b72 in 873ms, sequenceid=226, compaction requested=true 2023-04-17 08:00:31,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:31,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:31,243 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:00:31,243 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:00:31,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:31,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:31,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,245 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:31,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:31,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,245 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:31,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:31,246 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:31,246 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162708 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:00:31,246 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:31,247 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72558 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:00:31,247 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,247 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:31,247 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,247 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/bf911c7cf81b47e285c1386333208bb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=158.9 K 2023-04-17 08:00:31,247 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1e543368df194ccda99e5f8f3a4b0bcf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=70.9 K 2023-04-17 08:00:31,247 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting bf911c7cf81b47e285c1386333208bb5, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:31,247 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1e543368df194ccda99e5f8f3a4b0bcf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:31,248 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6151830fdb024b039e0a62efbf88fd0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681718424460 2023-04-17 08:00:31,248 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ee3eece6ce8f400d973336268c77c59b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681718424460 2023-04-17 08:00:31,248 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 645bcc51f0f34d9686bedb23d09e654c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681718425613 2023-04-17 08:00:31,248 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d3f084ad6de746ff85cf75109c23b0c4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681718425613 2023-04-17 08:00:31,249 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d144665de47e482e9ddb44014cd4e750, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681718427742 2023-04-17 08:00:31,249 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6f488f27d6624cf3959c6d811515f84f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681718427742 2023-04-17 08:00:31,249 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1288690b842248b2ac394c2c8b19f88d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681718428900 2023-04-17 08:00:31,249 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting da633a9c895a4c65a1bad6b53694c2b4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681718428900 2023-04-17 08:00:31,250 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting da0704e578314e6a8d7fc0777214d2c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430057 2023-04-17 08:00:31,250 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 766223a5285945ee805001459c4c23b4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430039 2023-04-17 08:00:31,265 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:31,281 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#165 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:31,287 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/13108bd949d5410cba43e99d97c520ce as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/13108bd949d5410cba43e99d97c520ce 2023-04-17 08:00:31,292 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into 13108bd949d5410cba43e99d97c520ce(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:31,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:31,292 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=10, startTime=1681718431243; duration=0sec 2023-04-17 08:00:31,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:31,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:00:31,294 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72558 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:00:31,294 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:31,294 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,294 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/404fa4426cbd4f9ca5f12b2a3a150a20, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=70.9 K 2023-04-17 08:00:31,294 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 404fa4426cbd4f9ca5f12b2a3a150a20, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681718424102 2023-04-17 08:00:31,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 09beb80d10d64e7c9ad04d9bc1c56493, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681718424460 2023-04-17 08:00:31,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 98b336de1bde4083b06af36d02422de4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681718425613 2023-04-17 08:00:31,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting cc7e80e7779c4351b5ef4e0e364c39e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681718427742 2023-04-17 08:00:31,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2a7037488c9d49e5a84264bab780ccd5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681718428900 2023-04-17 08:00:31,297 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6c2401bf114b4e3abd220e9e018f0373, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430057 2023-04-17 08:00:31,306 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#166 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:31,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/b65e848a8a4a4d329f4b84657d8cd13d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b65e848a8a4a4d329f4b84657d8cd13d 2023-04-17 08:00:31,324 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into b65e848a8a4a4d329f4b84657d8cd13d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:31,325 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:31,325 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=10, startTime=1681718431243; duration=0sec 2023-04-17 08:00:31,325 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:31,346 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:31,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:31,349 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:31,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:31,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:31,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,356 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:31,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:31,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:31,358 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:31,358 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:31,359 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:31,359 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:31,359 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:31,359 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:31,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:31,360 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:31,390 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f50120c0b9aa44a5bbfa80f16f2ce93c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f50120c0b9aa44a5bbfa80f16f2ce93c 2023-04-17 08:00:31,396 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b0cebbb3095d4a52a605e3b9562ec75c 2023-04-17 08:00:31,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:31,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:31,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718491569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:31,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718491672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,830 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/eda09650216243a886fcb4eb1ed82517 2023-04-17 08:00:31,846 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/1b63d12461d343abb44143c6d5c432bb 2023-04-17 08:00:31,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b0cebbb3095d4a52a605e3b9562ec75c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c 2023-04-17 08:00:31,859 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c, entries=150, sequenceid=240, filesize=25.3 K 2023-04-17 08:00:31,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/eda09650216243a886fcb4eb1ed82517 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517 2023-04-17 08:00:31,864 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517, entries=150, sequenceid=240, filesize=11.8 K 2023-04-17 08:00:31,865 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/1b63d12461d343abb44143c6d5c432bb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb 2023-04-17 08:00:31,869 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb, entries=150, sequenceid=240, filesize=11.8 K 2023-04-17 08:00:31,870 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f26affd8ee8c5b2a0826f19ce7699b72 in 510ms, sequenceid=240, compaction requested=false 2023-04-17 08:00:31,870 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:31,870 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:31,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:31,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:31,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:31,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:31,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718431553, max timestamp=1681718431877Num uniques 859; 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718431553, max timestamp=1681718431877Num uniques 859; 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718431553, max timestamp=1681718431877Num uniques 859; 2023-04-17 08:00:31,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:31,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-17 08:00:31,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:31,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:31,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:31,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:31,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:31,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:31,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:31,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@798c81b9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:31,906 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:31,906 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,915 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:31,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:31,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:31,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:31,931 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:31,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,931 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,931 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:31,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:31,937 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:31,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:31,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:31,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:31,937 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:31,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:31,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:31,945 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:31,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718431930, max timestamp=1681718431954Num uniques 859; 2023-04-17 08:00:31,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:31,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718431930, max timestamp=1681718431954Num uniques 859; 2023-04-17 08:00:31,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:31,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718431930, max timestamp=1681718431954Num uniques 859; 2023-04-17 08:00:31,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:31,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,986 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:31,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:31,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:31,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:31,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:31,986 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:31,986 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:31,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:31,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:31,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:31,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:31,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:31,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:31,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718431990, max timestamp=1681718432015Num uniques 1068; 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718431990, max timestamp=1681718432015Num uniques 1068; 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718431990, max timestamp=1681718432015Num uniques 1068; 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:00:32,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:00:32,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681718432058, max timestamp=1681718432074Num uniques 1027; 2023-04-17 08:00:32,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 08:00:32,086 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681718432058, max timestamp=1681718432074Num uniques 1027; 2023-04-17 08:00:32,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 08:00:32,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681718432058, max timestamp=1681718432074Num uniques 1027; 2023-04-17 08:00:32,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 08:00:32,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,096 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,128 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,128 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,128 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,129 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a35f546[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,141 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,148 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,156 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:32,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:32,158 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:32,158 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:32,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:32,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:32,159 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:32,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,166 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718432150, max timestamp=1681718432169Num uniques 1036; 2023-04-17 08:00:32,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 08:00:32,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718432150, max timestamp=1681718432169Num uniques 1036; 2023-04-17 08:00:32,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718432150, max timestamp=1681718432169Num uniques 1036; 2023-04-17 08:00:32,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 08:00:32,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:32,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,173 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,173 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718432217, max timestamp=1681718432235Num uniques 1045; 2023-04-17 08:00:32,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:00:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718432217, max timestamp=1681718432235Num uniques 1045; 2023-04-17 08:00:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718432217, max timestamp=1681718432235Num uniques 1045; 2023-04-17 08:00:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:00:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:00:32,274 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,276 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,276 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,297 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,297 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,297 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,297 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,306 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,307 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36106833[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,308 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,308 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718432269, max timestamp=1681718432311Num uniques 1104; 2023-04-17 08:00:32,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-17 08:00:32,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718432269, max timestamp=1681718432311Num uniques 1104; 2023-04-17 08:00:32,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-17 08:00:32,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681718432269, max timestamp=1681718432311Num uniques 1104; 2023-04-17 08:00:32,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-17 08:00:32,314 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,315 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,323 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,323 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:32,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:32,326 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:32,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:32,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:32,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:32,326 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:32,327 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041711f2618f58d243f88ef9b1318e660d4b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711f2618f58d243f88ef9b1318e660d4b 2023-04-17 08:00:32,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,341 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/ffb12697cb63419ea97b1223226a12b3 2023-04-17 08:00:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:32,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:32,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,348 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:32,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:32,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,348 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:32,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,349 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718432354, max timestamp=1681718432375Num uniques 1063; 2023-04-17 08:00:32,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-17 08:00:32,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681718432354, max timestamp=1681718432375Num uniques 1063; 2023-04-17 08:00:32,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681718432354, max timestamp=1681718432375Num uniques 1063; 2023-04-17 08:00:32,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-17 08:00:32,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-17 08:00:32,449 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,451 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,451 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,461 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,461 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681718432425, max timestamp=1681718432449Num uniques 1022; 2023-04-17 08:00:32,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-17 08:00:32,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681718432425, max timestamp=1681718432449Num uniques 1072; 2023-04-17 08:00:32,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681718432425, max timestamp=1681718432449Num uniques 1022; 2023-04-17 08:00:32,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-17 08:00:32,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-17 08:00:32,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b50a62a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,489 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,489 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,489 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:32,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:32,492 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:32,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:32,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:32,492 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:32,492 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:32,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,499 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,506 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,506 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,506 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:32,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:32,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,506 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,507 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:32,507 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:32,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718432503, max timestamp=1681718432530Num uniques 1031; 2023-04-17 08:00:32,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-17 08:00:32,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718432503, max timestamp=1681718432530Num uniques 1031; 2023-04-17 08:00:32,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718432503, max timestamp=1681718432530Num uniques 1031; 2023-04-17 08:00:32,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-17 08:00:32,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-17 08:00:32,547 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/5c98eb9de281478c8d207f6fc43656aa as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5c98eb9de281478c8d207f6fc43656aa 2023-04-17 08:00:32,554 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into 5c98eb9de281478c8d207f6fc43656aa(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:00:32,554 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:32,554 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=10, startTime=1681718431243; duration=1sec 2023-04-17 08:00:32,554 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:32,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681718432553, max timestamp=1681718432591Num uniques 1040; 2023-04-17 08:00:32,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-17 08:00:32,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681718432553, max timestamp=1681718432591Num uniques 1040; 2023-04-17 08:00:32,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-17 08:00:32,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681718432553, max timestamp=1681718432591Num uniques 1040; 2023-04-17 08:00:32,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-17 08:00:32,607 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,611 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,620 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,620 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,620 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,620 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,621 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,622 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,622 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,622 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,622 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,645 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,646 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,646 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718432629, max timestamp=1681718432647Num uniques 1049; 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718432629, max timestamp=1681718432647Num uniques 1049; 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718432629, max timestamp=1681718432647Num uniques 1049; 2023-04-17 08:00:32,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-17 08:00:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664c71b0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,654 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,654 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,673 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,681 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:32,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:32,684 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:32,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:32,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:32,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:32,684 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:32,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,691 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,692 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:32,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,704 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:32,704 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,703 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681718432692, max timestamp=1681718432725Num uniques 1108; 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681718432692, max timestamp=1681718432725Num uniques 1108; 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681718432692, max timestamp=1681718432725Num uniques 1108; 2023-04-17 08:00:32,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-17 08:00:32,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/3604952efa5946438a11e4387da667a9 2023-04-17 08:00:32,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681718432755, max timestamp=1681718432775Num uniques 1067; 2023-04-17 08:00:32,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-17 08:00:32,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681718432755, max timestamp=1681718432775Num uniques 1067; 2023-04-17 08:00:32,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-17 08:00:32,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681718432765, max timestamp=1681718432775Num uniques 1067; 2023-04-17 08:00:32,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-17 08:00:32,804 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,806 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,806 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,806 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,817 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,817 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,817 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,817 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,817 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,823 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,824 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231627fa[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,824 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,831 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681718432809, max timestamp=1681718432829Num uniques 1026; 2023-04-17 08:00:32,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-17 08:00:32,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681718432809, max timestamp=1681718432829Num uniques 1076; 2023-04-17 08:00:32,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681718432809, max timestamp=1681718432829Num uniques 1076; 2023-04-17 08:00:32,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-17 08:00:32,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-17 08:00:32,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,839 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:32,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,841 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:32,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:32,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:32,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:32,841 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:32,848 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,848 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,848 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,856 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,856 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:32,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,857 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:32,857 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:32,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:32,857 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:32,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681718432891, max timestamp=1681718432898Num uniques 1035; 2023-04-17 08:00:32,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-17 08:00:32,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681718432891, max timestamp=1681718432898Num uniques 1035; 2023-04-17 08:00:32,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681718432891, max timestamp=1681718432898Num uniques 1035; 2023-04-17 08:00:32,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-17 08:00:32,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-17 08:00:32,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681718432936, max timestamp=1681718432945Num uniques 1044; 2023-04-17 08:00:32,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-17 08:00:32,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681718432936, max timestamp=1681718432945Num uniques 1044; 2023-04-17 08:00:32,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-17 08:00:32,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681718432936, max timestamp=1681718432945Num uniques 1044; 2023-04-17 08:00:32,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-17 08:00:32,957 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:32,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:32,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:32,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:32,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:32,960 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:32,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:32,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:32,960 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,961 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:32,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:32,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:32,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:32,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:32,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:32,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:32,971 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:32,971 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:32,971 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:32,971 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:32,971 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:32,971 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,971 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:32,972 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:32,972 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:32,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:32,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:32,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:32,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:32,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,981 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:32,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f2925b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:32,983 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:32,983 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,989 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:32,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:32,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:32,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:32,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681718432974, max timestamp=1681718432994Num uniques 1053; 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681718432974, max timestamp=1681718432994Num uniques 1053; 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681718432974, max timestamp=1681718432994Num uniques 1053; 2023-04-17 08:00:32,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-17 08:00:32,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:32,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:32,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:32,998 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:33,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:33,000 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:33,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:33,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:33,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:33,000 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:33,006 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,006 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,006 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:33,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:33,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,015 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,015 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,015 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681718433035, max timestamp=1681718433063Num uniques 1062; 2023-04-17 08:00:33,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681718433035, max timestamp=1681718433063Num uniques 1062; 2023-04-17 08:00:33,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-17 08:00:33,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681718433035, max timestamp=1681718433063Num uniques 1062; 2023-04-17 08:00:33,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-17 08:00:33,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718433093, max timestamp=1681718433106Num uniques 1121; 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718433093, max timestamp=1681718433106Num uniques 1121; 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681718433093, max timestamp=1681718433106Num uniques 1121; 2023-04-17 08:00:33,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-17 08:00:33,115 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:33,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:33,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:33,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:33,118 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:33,118 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,118 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:33,118 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:33,118 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,118 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:33,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:33,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:33,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:33,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:33,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:33,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:33,129 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,130 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:33,130 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:33,130 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:33,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:33,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:33,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:33,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:33,141 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:33,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ed0759[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:33,141 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:33,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:33,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:33,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681718433146, max timestamp=1681718433160Num uniques 1080; 2023-04-17 08:00:33,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-17 08:00:33,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681718433146, max timestamp=1681718433160Num uniques 1080; 2023-04-17 08:00:33,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-17 08:00:33,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:33,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681718433146, max timestamp=1681718433160Num uniques 1080; 2023-04-17 08:00:33,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-17 08:00:33,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/0c4bcd2a85ae42a288d9169ad1702581 2023-04-17 08:00:33,173 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:33,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/ffb12697cb63419ea97b1223226a12b3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3 2023-04-17 08:00:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:33,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:33,177 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:33,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:33,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:33,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:33,177 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:33,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3, entries=200, sequenceid=261, filesize=32.3 K 2023-04-17 08:00:33,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/3604952efa5946438a11e4387da667a9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9 2023-04-17 08:00:33,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9, entries=150, sequenceid=261, filesize=11.9 K 2023-04-17 08:00:33,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/0c4bcd2a85ae42a288d9169ad1702581 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,199 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:33,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:33,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:33,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,199 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:33,199 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:33,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581, entries=150, sequenceid=261, filesize=11.9 K 2023-04-17 08:00:33,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=114.05 KB/116790 for f26affd8ee8c5b2a0826f19ce7699b72 in 1320ms, sequenceid=261, compaction requested=true 2023-04-17 08:00:33,208 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:33,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:33,208 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:33,208 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:33,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:33,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:33,209 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85374 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:33,209 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:33,210 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,210 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5c98eb9de281478c8d207f6fc43656aa, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=83.4 K 2023-04-17 08:00:33,209 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:33,210 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:33,210 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,210 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/13108bd949d5410cba43e99d97c520ce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=35.9 K 2023-04-17 08:00:33,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=120.76 KB heapSize=343.17 KB 2023-04-17 08:00:33,210 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5c98eb9de281478c8d207f6fc43656aa, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430057 2023-04-17 08:00:33,211 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 13108bd949d5410cba43e99d97c520ce, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430057 2023-04-17 08:00:33,211 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b0cebbb3095d4a52a605e3b9562ec75c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681718430393 2023-04-17 08:00:33,211 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting eda09650216243a886fcb4eb1ed82517, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681718430393 2023-04-17 08:00:33,211 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ffb12697cb63419ea97b1223226a12b3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431553 2023-04-17 08:00:33,211 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3604952efa5946438a11e4387da667a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431566 2023-04-17 08:00:33,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:33,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:33,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:33,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:33,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:33,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:33,219 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:33,248 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#174 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:33,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:33,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 586 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718493251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,277 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/5adc5efdf14a4a118f147314ba624a8f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5adc5efdf14a4a118f147314ba624a8f 2023-04-17 08:00:33,284 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into 5adc5efdf14a4a118f147314ba624a8f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:33,284 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:33,284 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=13, startTime=1681718433208; duration=0sec 2023-04-17 08:00:33,284 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:33,284 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:33,286 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:33,286 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:33,286 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,286 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b65e848a8a4a4d329f4b84657d8cd13d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=35.9 K 2023-04-17 08:00:33,286 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b65e848a8a4a4d329f4b84657d8cd13d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1681718430057 2023-04-17 08:00:33,287 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1b63d12461d343abb44143c6d5c432bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681718430393 2023-04-17 08:00:33,287 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0c4bcd2a85ae42a288d9169ad1702581, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431566 2023-04-17 08:00:33,300 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:33,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:33,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:33,317 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#176 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:33,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:33,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:33,325 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:33,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:33,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:33,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:33,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:33,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:33,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:33,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:33,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:33,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:33,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:33,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:33,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:33,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:33,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:33,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:33,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:33,356 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,357 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:33,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d293a05[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:33,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:33,361 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/492d0d030a264bf1b3e2526fbc21d62d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/492d0d030a264bf1b3e2526fbc21d62d 2023-04-17 08:00:33,361 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:33,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 588 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718493359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,368 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into 492d0d030a264bf1b3e2526fbc21d62d(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:33,368 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:33,368 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=13, startTime=1681718433208; duration=0sec 2023-04-17 08:00:33,369 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:33,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:33,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:33,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:33,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:33,378 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:33,378 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,378 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:33,379 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:33,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:33,382 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:33,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:33,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:33,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:33,382 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:33,389 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,389 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:33,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:33,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,398 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,398 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:33,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,399 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,499 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:33,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:33,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:33,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:33,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:33,501 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:33,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:33,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:33,502 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,502 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,511 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,511 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:33,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:33,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:33,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:33,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:33,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:33,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:33,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:33,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f12e9ad[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:33,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:33,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:33,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:33,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:33,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:33,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:33,539 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:33,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:33,541 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:33,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:33,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:33,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:33,541 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:33,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:33,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:33,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:33,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,556 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,557 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,557 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:33,557 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:33,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:33,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 590 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718493565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,657 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:33,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:33,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:33,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:33,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:33,661 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:33,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:33,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:33,664 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,664 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,731 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417074877402ae949d08316c320f24a303a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417074877402ae949d08316c320f24a303a 2023-04-17 08:00:33,742 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/6788d458c3e442949acce32a47b5c6de as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6788d458c3e442949acce32a47b5c6de 2023-04-17 08:00:33,746 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into 6788d458c3e442949acce32a47b5c6de(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:33,746 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:33,746 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=13, startTime=1681718433208; duration=0sec 2023-04-17 08:00:33,746 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:33,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:33,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:33,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:33,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:33,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:33,816 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,816 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:33,816 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:33,816 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:33,816 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:33,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,831 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,832 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,832 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:33,833 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df0cc5e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:33,833 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,839 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:33,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:33,840 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:33,840 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:33,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:33,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:33,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:33,866 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:33,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:33,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:33,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:33,866 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:33,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:33,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 592 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718493869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:33,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:33,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:33,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:33,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:33,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:33,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:33,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:33,890 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:33,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:33,890 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:33,890 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:33,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:33,990 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:33,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:33,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:33,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:33,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:33,993 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:33,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:33,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:33,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:33,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:33,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,003 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,003 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,017 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,017 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,023 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,023 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,023 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d27cb76[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,024 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,031 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,041 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,041 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,056 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,057 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,057 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,136 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=623, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b6d9a7d8e255491b926402dde5f7c9f1 2023-04-17 08:00:34,157 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,159 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,171 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,171 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,171 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,171 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,181 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59fa3fc5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,207 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,209 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,220 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,220 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,321 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,326 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,326 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,336 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,338 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,338 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,348 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,348 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,348 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,348 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@314dfad6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,349 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,356 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,364 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,364 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,364 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,365 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,367 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,367 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,367 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:34,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 594 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718494377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,398 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,398 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,498 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,501 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,514 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,514 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,515 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,516 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,516 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,516 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,516 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,516 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,523 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa4529d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,531 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,540 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,542 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,542 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/6a98d50f35fe47199fc0772af59432f3 2023-04-17 08:00:34,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,548 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,561 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,561 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,562 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,562 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,662 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,665 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,674 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,674 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,675 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,675 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,675 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,675 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,675 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,675 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,675 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,681 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@91b5cc4[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,689 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,698 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,700 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,700 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,708 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,715 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,715 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,715 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,815 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,818 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,829 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:34,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:34,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:34,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:34,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:34,830 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,830 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:34,830 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:34,830 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:34,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:34,839 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,840 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,840 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:34,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1175e35[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:34,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:34,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,848 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:34,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:34,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:34,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:34,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:34,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,856 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:34,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:34,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:34,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:34,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:34,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:34,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:34,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,865 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:34,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:34,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:34,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:34,896 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:34,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:34,896 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:34,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:34,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:34,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/b17119893e574e9086aac8ef23bc4f1e 2023-04-17 08:00:34,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/b6d9a7d8e255491b926402dde5f7c9f1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1 2023-04-17 08:00:34,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1, entries=150, sequenceid=623, filesize=25.4 K 2023-04-17 08:00:34,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/6a98d50f35fe47199fc0772af59432f3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3 2023-04-17 08:00:34,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3, entries=150, sequenceid=623, filesize=11.9 K 2023-04-17 08:00:34,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/b17119893e574e9086aac8ef23bc4f1e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e 2023-04-17 08:00:34,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e, entries=150, sequenceid=623, filesize=11.9 K 2023-04-17 08:00:34,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~343.13 KB/351360, currentSize=60.38 KB/61830 for f26affd8ee8c5b2a0826f19ce7699b72 in 1765ms, sequenceid=623, compaction requested=false 2023-04-17 08:00:34,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:34,996 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:34,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:34,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:34,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:34,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:34,999 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:34,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:34,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:34,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:34,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:34,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:35,011 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:35,011 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:35,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:35,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:35,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:35,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:35,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:35,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:35,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:35,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:35,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:35,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:35,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:35,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:35,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:35,013 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:35,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:35,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:35,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:35,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:35,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:35,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:35,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:35,425 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177e1e50dc8a694bd084c6f729d748341b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177e1e50dc8a694bd084c6f729d748341b 2023-04-17 08:00:35,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:35,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 613 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718495426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:35,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:35,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 615 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718495527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:35,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:35,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 617 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718495730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:35,832 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=638, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/e0c1530a73ce4b8282935eb16fafad68 2023-04-17 08:00:36,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:36,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 619 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718496032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,242 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/98109ad402a04633b896dfc944a4baee 2023-04-17 08:00:36,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:36,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 621 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718496534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,652 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/8a182018b4f34094be83c59082545335 2023-04-17 08:00:36,657 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/e0c1530a73ce4b8282935eb16fafad68 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68 2023-04-17 08:00:36,660 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68, entries=150, sequenceid=638, filesize=25.4 K 2023-04-17 08:00:36,661 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/98109ad402a04633b896dfc944a4baee as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee 2023-04-17 08:00:36,665 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee, entries=150, sequenceid=638, filesize=11.9 K 2023-04-17 08:00:36,666 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/8a182018b4f34094be83c59082545335 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335 2023-04-17 08:00:36,673 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335, entries=150, sequenceid=638, filesize=11.9 K 2023-04-17 08:00:36,673 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f26affd8ee8c5b2a0826f19ce7699b72 in 1660ms, sequenceid=638, compaction requested=true 2023-04-17 08:00:36,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:36,673 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:36,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:36,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:36,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:36,674 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:36,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:36,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:36,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:36,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd29f8b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:36,708 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:36,708 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:36,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:36,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:36,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:36,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:36,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:36,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:36,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:36,725 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:36,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:36,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:36,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:36,725 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:36,731 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,731 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:36,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,731 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:36,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:36,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:36,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:36,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:36,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:36,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:36,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:36,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:36,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:36,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:36,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,740 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:36,740 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:36,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:36,740 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:36,840 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:36,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:36,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:36,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:36,843 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:36,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:36,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:36,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:36,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:36,853 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:36,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:36,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:36,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:36,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:36,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:36,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:36,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:36,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:36,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:36,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:36,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:36,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:36,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:36,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:36,855 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:36,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:36,867 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417419ac54f216b4450b404d63c4742f556 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417419ac54f216b4450b404d63c4742f556 2023-04-17 08:00:37,272 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=658, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/04d9e4184dfd43878a4e564f27dee424 2023-04-17 08:00:37,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:37,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:37,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:37,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 632 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718497563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:37,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 634 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718497665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,683 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/296ad9ab6a3e46be88ac98a15f948ab1 2023-04-17 08:00:37,693 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/6abd5a44dfc74753bd53e709082b6595 2023-04-17 08:00:37,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/04d9e4184dfd43878a4e564f27dee424 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424 2023-04-17 08:00:37,699 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424, entries=150, sequenceid=658, filesize=25.4 K 2023-04-17 08:00:37,700 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/296ad9ab6a3e46be88ac98a15f948ab1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1 2023-04-17 08:00:37,704 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1, entries=150, sequenceid=658, filesize=11.9 K 2023-04-17 08:00:37,704 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/6abd5a44dfc74753bd53e709082b6595 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595 2023-04-17 08:00:37,707 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595, entries=150, sequenceid=658, filesize=11.9 K 2023-04-17 08:00:37,708 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f26affd8ee8c5b2a0826f19ce7699b72 in 853ms, sequenceid=658, compaction requested=true 2023-04-17 08:00:37,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:37,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:37,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:37,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:37,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:37,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:37,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:37,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378a0e56[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:37,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:37,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,756 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:37,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:37,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:37,756 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,756 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:37,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:37,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:37,764 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:37,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:37,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:37,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:37,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:37,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:37,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:37,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:37,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:37,781 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:37,781 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:37,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,781 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:37,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:37,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,782 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:37,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:37,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:37,882 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:37,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:37,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:37,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:37,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:37,884 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:37,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:37,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:37,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:37,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,885 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:37,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:37,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:37,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:37,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:37,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:37,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:37,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:37,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:37,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:37,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:37,896 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:37,896 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:37,896 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:37,896 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:37,906 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6424991f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:37,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:37,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:37,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:37,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:37,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:37,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:37,923 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,923 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:37,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:37,924 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:37,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:37,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:37,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:37,924 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:37,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:37,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 653 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718497922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:37,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:37,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:37,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:37,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:37,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:37,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:37,940 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:37,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,940 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:37,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:37,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:37,940 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:38,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 655 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718498027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,040 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:38,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:38,042 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:38,042 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,042 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:38,043 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:38,043 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,043 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:38,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:38,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:38,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:38,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:38,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:38,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:38,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:38,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:38,073 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:38,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:38,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:38,073 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:38,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff99558[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:38,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:38,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,089 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,089 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:38,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:38,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:38,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:38,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:38,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,098 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,098 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:38,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:38,099 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:38,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:38,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:38,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:38,100 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:38,106 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,106 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,106 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:38,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:38,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,115 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:38,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,115 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:38,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,115 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,218 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:38,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:38,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:38,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:38,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:38,221 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:38,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:38,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:38,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:38,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 657 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718498230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,231 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:38,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:38,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:38,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:38,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:38,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:38,233 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:38,233 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:38,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:38,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:38,233 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:38,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,240 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149c67eb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:38,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:38,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,248 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,248 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,248 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:38,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:38,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:38,248 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:38,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:38,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:38,258 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:38,258 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:38,258 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:38,258 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:38,258 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:38,264 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,264 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,264 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:38,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:38,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:38,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,273 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:38,273 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,273 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,283 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417da93c3d890b145caa428437aa637f8c1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da93c3d890b145caa428437aa637f8c1 2023-04-17 08:00:38,373 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:38,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:38,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:38,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:38,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:38,376 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:38,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:38,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:38,377 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,377 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,386 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,386 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:38,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:38,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:38,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. started... 2023-04-17 08:00:38,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. as already flushing 2023-04-17 08:00:38,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:38,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:38,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,398 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:38,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73833390[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:38,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:38,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,406 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:38,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:38,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:38,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:38,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:38,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:38,417 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:38,417 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:38,417 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:38,417 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:38,417 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:38,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:38,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:38,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:38,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:38,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:38,431 DEBUG [Thread-920] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:38,432 DEBUG [Thread-920] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:38,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:38,451 DEBUG [Thread-921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:62477 2023-04-17 08:00:38,451 DEBUG [Thread-921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:38,454 DEBUG [Thread-923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52902392 to 127.0.0.1:62477 2023-04-17 08:00:38,454 DEBUG [Thread-923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:38,454 DEBUG [Thread-925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e1da5ba to 127.0.0.1:62477 2023-04-17 08:00:38,454 DEBUG [Thread-925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:38,458 DEBUG [Thread-927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17a1bd04 to 127.0.0.1:62477 2023-04-17 08:00:38,458 DEBUG [Thread-927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:38,532 DEBUG [Thread-920] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:38,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:38,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 659 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718498532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:38,692 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=671, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6b20ffcd233f43dc933d89d0538e5234 2023-04-17 08:00:39,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:39,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 661 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:55280 deadline: 1681718499035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:39,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/e5b0d1b6951b400eb80d0194c4aba862 2023-04-17 08:00:39,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:39,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55310 deadline: 1681718499206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:39,208 DEBUG [Thread-916] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:39,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:39,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55290 deadline: 1681718499214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:39,216 DEBUG [Thread-910] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:39,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:39,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55294 deadline: 1681718499296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:39,297 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18262 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:39,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:55242 deadline: 1681718499304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:39,305 DEBUG [Thread-914] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18269 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:00:39,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/62b51e97cb204df0a785f5e8ded75714 2023-04-17 08:00:39,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/6b20ffcd233f43dc933d89d0538e5234 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234 2023-04-17 08:00:39,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234, entries=200, sequenceid=671, filesize=32.3 K 2023-04-17 08:00:39,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/e5b0d1b6951b400eb80d0194c4aba862 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862 2023-04-17 08:00:39,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862, entries=150, sequenceid=671, filesize=11.9 K 2023-04-17 08:00:39,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/62b51e97cb204df0a785f5e8ded75714 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714 2023-04-17 08:00:39,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714, entries=150, sequenceid=671, filesize=11.9 K 2023-04-17 08:00:39,550 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f26affd8ee8c5b2a0826f19ce7699b72 in 1679ms, sequenceid=671, compaction requested=true 2023-04-17 08:00:39,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:39,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:39,550 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:00:39,550 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:00:39,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:39,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:39,551 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/A is initiating minor compaction (all files) 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/B is initiating minor compaction (all files) 2023-04-17 08:00:39,552 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/A in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:39,552 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/B in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:39,552 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/492d0d030a264bf1b3e2526fbc21d62d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=134.7 K 2023-04-17 08:00:39,552 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5adc5efdf14a4a118f147314ba624a8f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=60.1 K 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 492d0d030a264bf1b3e2526fbc21d62d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431566 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5adc5efdf14a4a118f147314ba624a8f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431566 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b6d9a7d8e255491b926402dde5f7c9f1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1681718433193 2023-04-17 08:00:39,552 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6a98d50f35fe47199fc0772af59432f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1681718433193 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e0c1530a73ce4b8282935eb16fafad68, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681718433239 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 98109ad402a04633b896dfc944a4baee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681718433239 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 04d9e4184dfd43878a4e564f27dee424, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681718435400 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 296ad9ab6a3e46be88ac98a15f948ab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681718435400 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6b20ffcd233f43dc933d89d0538e5234, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681718437558 2023-04-17 08:00:39,553 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e5b0d1b6951b400eb80d0194c4aba862, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681718437558 2023-04-17 08:00:39,561 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#B#compaction#188 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:39,567 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#A#compaction#189 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:39,574 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/3a109ac01a2c4639b7f0f4c1996fd3ab as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3a109ac01a2c4639b7f0f4c1996fd3ab 2023-04-17 08:00:39,578 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/B of f26affd8ee8c5b2a0826f19ce7699b72 into 3a109ac01a2c4639b7f0f4c1996fd3ab(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:39,578 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:39,578 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/B, priority=11, startTime=1681718439550; duration=0sec 2023-04-17 08:00:39,578 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:39,578 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:00:39,580 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:00:39,580 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): f26affd8ee8c5b2a0826f19ce7699b72/C is initiating minor compaction (all files) 2023-04-17 08:00:39,580 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f26affd8ee8c5b2a0826f19ce7699b72/C in TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:39,580 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6788d458c3e442949acce32a47b5c6de, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp, totalSize=60.1 K 2023-04-17 08:00:39,580 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6788d458c3e442949acce32a47b5c6de, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681718431566 2023-04-17 08:00:39,581 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b17119893e574e9086aac8ef23bc4f1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1681718433193 2023-04-17 08:00:39,581 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8a182018b4f34094be83c59082545335, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681718433239 2023-04-17 08:00:39,581 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6abd5a44dfc74753bd53e709082b6595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681718435400 2023-04-17 08:00:39,582 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 62b51e97cb204df0a785f5e8ded75714, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681718437558 2023-04-17 08:00:39,589 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): f26affd8ee8c5b2a0826f19ce7699b72#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:39,999 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/b39aeac1c7634c56b414ab053afd70c4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b39aeac1c7634c56b414ab053afd70c4 2023-04-17 08:00:40,006 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/C of f26affd8ee8c5b2a0826f19ce7699b72 into b39aeac1c7634c56b414ab053afd70c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:40,006 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:40,006 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/C, priority=11, startTime=1681718439550; duration=0sec 2023-04-17 08:00:40,006 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:40,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:40,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:40,044 DEBUG [Thread-912] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x595355a0 to 127.0.0.1:62477 2023-04-17 08:00:40,044 DEBUG [Thread-912] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:40,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:40,463 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417232c74b538cf4b09ad0181a2fe749d26 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417232c74b538cf4b09ad0181a2fe749d26 2023-04-17 08:00:40,807 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/d108abbedc214670a35aeae0f832b5f4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d108abbedc214670a35aeae0f832b5f4 2023-04-17 08:00:40,814 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f26affd8ee8c5b2a0826f19ce7699b72/A of f26affd8ee8c5b2a0826f19ce7699b72 into d108abbedc214670a35aeae0f832b5f4(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:00:40,815 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:40,815 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72., storeName=f26affd8ee8c5b2a0826f19ce7699b72/A, priority=11, startTime=1681718439550; duration=1sec 2023-04-17 08:00:40,815 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:40,871 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=693, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/e6636bf17d444de39d111881adbb15c9 2023-04-17 08:00:41,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/33758a27f2ee4e1aaee1e9fa01b7e588 2023-04-17 08:00:41,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/30d81ee3580c4362ae304648a81333f7 2023-04-17 08:00:41,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/e6636bf17d444de39d111881adbb15c9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e6636bf17d444de39d111881adbb15c9 2023-04-17 08:00:41,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e6636bf17d444de39d111881adbb15c9, entries=150, sequenceid=693, filesize=25.4 K 2023-04-17 08:00:41,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/33758a27f2ee4e1aaee1e9fa01b7e588 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/33758a27f2ee4e1aaee1e9fa01b7e588 2023-04-17 08:00:41,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/33758a27f2ee4e1aaee1e9fa01b7e588, entries=150, sequenceid=693, filesize=11.9 K 2023-04-17 08:00:41,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/30d81ee3580c4362ae304648a81333f7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/30d81ee3580c4362ae304648a81333f7 2023-04-17 08:00:41,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/30d81ee3580c4362ae304648a81333f7, entries=150, sequenceid=693, filesize=11.9 K 2023-04-17 08:00:41,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for f26affd8ee8c5b2a0826f19ce7699b72 in 1698ms, sequenceid=693, compaction requested=false 2023-04-17 08:00:41,741 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:46,176 DEBUG [regionserver/jenkins-hbase16:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-17 08:00:49,225 DEBUG [Thread-916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a84dba6 to 127.0.0.1:62477 2023-04-17 08:00:49,225 DEBUG [Thread-916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:49,309 DEBUG [Thread-910] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x389356cd to 127.0.0.1:62477 2023-04-17 08:00:49,310 DEBUG [Thread-910] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:49,385 DEBUG [Thread-914] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38da9522 to 127.0.0.1:62477 2023-04-17 08:00:49,385 DEBUG [Thread-914] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:49,385 DEBUG [Thread-918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c0b7748 to 127.0.0.1:62477 2023-04-17 08:00:49,386 DEBUG [Thread-918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:49,386 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:00:49,386 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-17 08:00:49,386 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 554 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 6015 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5952 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2587 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7758 rows 2023-04-17 08:00:49,387 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2585 2023-04-17 08:00:49,388 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7755 rows 2023-04-17 08:00:49,388 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:00:49,388 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x556231fa to 127.0.0.1:62477 2023-04-17 08:00:49,388 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:00:49,390 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:00:49,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:00:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:49,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:49,396 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718449395"}]},"ts":"1681718449395"} 2023-04-17 08:00:49,396 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:00:49,411 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:00:49,412 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, UNASSIGN}] 2023-04-17 08:00:49,413 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, UNASSIGN 2023-04-17 08:00:49,413 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:49,414 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718449413"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718449413"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718449413"}]},"ts":"1681718449413"} 2023-04-17 08:00:49,415 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:49,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:49,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing f26affd8ee8c5b2a0826f19ce7699b72, disabling compactions & flushes 2023-04-17 08:00:49,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. after waiting 0 ms 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:49,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing f26affd8ee8c5b2a0826f19ce7699b72 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=A 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=B 2023-04-17 08:00:49,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:49,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f26affd8ee8c5b2a0826f19ce7699b72, store=C 2023-04-17 08:00:49,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:49,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:49,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ba7978b8cd484b4b8939ce1434e0aa26 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ba7978b8cd484b4b8939ce1434e0aa26 2023-04-17 08:00:50,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:50,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=701, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/00246ecc3b534691832cecf19d2217bc 2023-04-17 08:00:50,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:50,820 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/700fb7af873f44768d2688727cfbebe3 2023-04-17 08:00:51,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/f1bdc48075e24beeb8069fb8064dcbbc 2023-04-17 08:00:51,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/A/00246ecc3b534691832cecf19d2217bc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/00246ecc3b534691832cecf19d2217bc 2023-04-17 08:00:51,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/00246ecc3b534691832cecf19d2217bc, entries=150, sequenceid=701, filesize=25.4 K 2023-04-17 08:00:51,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/B/700fb7af873f44768d2688727cfbebe3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/700fb7af873f44768d2688727cfbebe3 2023-04-17 08:00:51,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/700fb7af873f44768d2688727cfbebe3, entries=150, sequenceid=701, filesize=11.9 K 2023-04-17 08:00:51,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/.tmp/C/f1bdc48075e24beeb8069fb8064dcbbc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/f1bdc48075e24beeb8069fb8064dcbbc 2023-04-17 08:00:51,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/f1bdc48075e24beeb8069fb8064dcbbc, entries=150, sequenceid=701, filesize=11.9 K 2023-04-17 08:00:51,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for f26affd8ee8c5b2a0826f19ce7699b72 in 1692ms, sequenceid=701, compaction requested=true 2023-04-17 08:00:51,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3a327a2b060a42c0a1f21d27c7111614, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/017c7d892b3241c3824bc8dbf4c6c88c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/bf911c7cf81b47e285c1386333208bb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5c98eb9de281478c8d207f6fc43656aa, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/492d0d030a264bf1b3e2526fbc21d62d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234] to archive 2023-04-17 08:00:51,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:51,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/17686e608318409e95e4f594bd5ba60b 2023-04-17 08:00:51,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/4f1742ca91a546d1b8037a1f1cbade19 2023-04-17 08:00:51,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d0c73298996e4d319db1ea0992b3bc8b 2023-04-17 08:00:51,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3a327a2b060a42c0a1f21d27c7111614 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3a327a2b060a42c0a1f21d27c7111614 2023-04-17 08:00:51,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6a964c0a171549f1b071681432f5d8a1 2023-04-17 08:00:51,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da465b55a5c14f969a6865a908133d06 2023-04-17 08:00:51,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/3fbd8aaf16224c879a927b9d70e91f84 2023-04-17 08:00:51,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/017c7d892b3241c3824bc8dbf4c6c88c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/017c7d892b3241c3824bc8dbf4c6c88c 2023-04-17 08:00:51,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5a4008b98e514331a0d50ef21efc3c14 2023-04-17 08:00:51,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/bf911c7cf81b47e285c1386333208bb5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/bf911c7cf81b47e285c1386333208bb5 2023-04-17 08:00:51,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b77e139c1599452a89f19bda8273c0d0 2023-04-17 08:00:51,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ee3eece6ce8f400d973336268c77c59b 2023-04-17 08:00:51,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d3f084ad6de746ff85cf75109c23b0c4 2023-04-17 08:00:51,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6f488f27d6624cf3959c6d811515f84f 2023-04-17 08:00:51,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/da633a9c895a4c65a1bad6b53694c2b4 2023-04-17 08:00:51,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/766223a5285945ee805001459c4c23b4 2023-04-17 08:00:51,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5c98eb9de281478c8d207f6fc43656aa to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/5c98eb9de281478c8d207f6fc43656aa 2023-04-17 08:00:51,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b0cebbb3095d4a52a605e3b9562ec75c 2023-04-17 08:00:51,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/ffb12697cb63419ea97b1223226a12b3 2023-04-17 08:00:51,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/492d0d030a264bf1b3e2526fbc21d62d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/492d0d030a264bf1b3e2526fbc21d62d 2023-04-17 08:00:51,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/b6d9a7d8e255491b926402dde5f7c9f1 2023-04-17 08:00:51,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e0c1530a73ce4b8282935eb16fafad68 2023-04-17 08:00:51,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/04d9e4184dfd43878a4e564f27dee424 2023-04-17 08:00:51,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/6b20ffcd233f43dc933d89d0538e5234 2023-04-17 08:00:51,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/c67223440fb1482aa95633dc3890a980, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/4ace0f5dcb314a9aa22881a23096de9f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1e543368df194ccda99e5f8f3a4b0bcf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/13108bd949d5410cba43e99d97c520ce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5adc5efdf14a4a118f147314ba624a8f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862] to archive 2023-04-17 08:00:51,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:51,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d86368bc42a046a791cbdd9fc6c2c6f5 2023-04-17 08:00:51,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/fee8e47ab49e48689f0a926ca039fbae 2023-04-17 08:00:51,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/c67223440fb1482aa95633dc3890a980 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/c67223440fb1482aa95633dc3890a980 2023-04-17 08:00:51,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/0a62cc5b22ad4475aaa728db81008722 2023-04-17 08:00:51,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/f3785a9ca9914cac8179d24536d306a0 2023-04-17 08:00:51,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/8a60fcb6cc2545c9a22a0aea16c17727 2023-04-17 08:00:51,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/4ace0f5dcb314a9aa22881a23096de9f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/4ace0f5dcb314a9aa22881a23096de9f 2023-04-17 08:00:51,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/46aebfd4101340129c2fefa7ec530044 2023-04-17 08:00:51,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/ce095c5bbbe74afe8f75abb832eb4f02 2023-04-17 08:00:51,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1e543368df194ccda99e5f8f3a4b0bcf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1e543368df194ccda99e5f8f3a4b0bcf 2023-04-17 08:00:51,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5c541f96c7264e6ab25d28bf63b0cc38 2023-04-17 08:00:51,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6151830fdb024b039e0a62efbf88fd0a 2023-04-17 08:00:51,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/645bcc51f0f34d9686bedb23d09e654c 2023-04-17 08:00:51,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/d144665de47e482e9ddb44014cd4e750 2023-04-17 08:00:51,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/1288690b842248b2ac394c2c8b19f88d 2023-04-17 08:00:51,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/13108bd949d5410cba43e99d97c520ce to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/13108bd949d5410cba43e99d97c520ce 2023-04-17 08:00:51,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/da0704e578314e6a8d7fc0777214d2c1 2023-04-17 08:00:51,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/eda09650216243a886fcb4eb1ed82517 2023-04-17 08:00:51,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5adc5efdf14a4a118f147314ba624a8f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/5adc5efdf14a4a118f147314ba624a8f 2023-04-17 08:00:51,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3604952efa5946438a11e4387da667a9 2023-04-17 08:00:51,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/6a98d50f35fe47199fc0772af59432f3 2023-04-17 08:00:51,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/98109ad402a04633b896dfc944a4baee 2023-04-17 08:00:51,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/296ad9ab6a3e46be88ac98a15f948ab1 2023-04-17 08:00:51,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/e5b0d1b6951b400eb80d0194c4aba862 2023-04-17 08:00:51,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/99cc1d8129f4448d850f4f5893a3b1b2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b1f3dcca28b14c39a0b832768f27363e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/404fa4426cbd4f9ca5f12b2a3a150a20, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b65e848a8a4a4d329f4b84657d8cd13d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6788d458c3e442949acce32a47b5c6de, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714] to archive 2023-04-17 08:00:51,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:00:51,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/3c30c3c1e852478687a65859e34117a4 2023-04-17 08:00:51,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/a13b4f41fcef4aacb4e0e046a1723a57 2023-04-17 08:00:51,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/99cc1d8129f4448d850f4f5893a3b1b2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/99cc1d8129f4448d850f4f5893a3b1b2 2023-04-17 08:00:51,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/15c8f871beba4c7a8762101bc7478330 2023-04-17 08:00:51,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/44552e23d0f8447fabd7ad6543a5de1d 2023-04-17 08:00:51,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/061e48e6773b473fbf911dae2f890aa2 2023-04-17 08:00:51,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b1f3dcca28b14c39a0b832768f27363e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b1f3dcca28b14c39a0b832768f27363e 2023-04-17 08:00:51,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1f10e0fcff4b4f87bd73621ca22287ae 2023-04-17 08:00:51,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/10ff008de8624839b4a6e9845c705e62 2023-04-17 08:00:51,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/404fa4426cbd4f9ca5f12b2a3a150a20 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/404fa4426cbd4f9ca5f12b2a3a150a20 2023-04-17 08:00:51,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/14865b668d5448f3bccac4124d8324e6 2023-04-17 08:00:51,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/09beb80d10d64e7c9ad04d9bc1c56493 2023-04-17 08:00:51,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/98b336de1bde4083b06af36d02422de4 2023-04-17 08:00:51,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/cc7e80e7779c4351b5ef4e0e364c39e6 2023-04-17 08:00:51,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/2a7037488c9d49e5a84264bab780ccd5 2023-04-17 08:00:51,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b65e848a8a4a4d329f4b84657d8cd13d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b65e848a8a4a4d329f4b84657d8cd13d 2023-04-17 08:00:51,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6c2401bf114b4e3abd220e9e018f0373 2023-04-17 08:00:51,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/1b63d12461d343abb44143c6d5c432bb 2023-04-17 08:00:51,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6788d458c3e442949acce32a47b5c6de to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6788d458c3e442949acce32a47b5c6de 2023-04-17 08:00:51,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/0c4bcd2a85ae42a288d9169ad1702581 2023-04-17 08:00:51,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b17119893e574e9086aac8ef23bc4f1e 2023-04-17 08:00:51,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/8a182018b4f34094be83c59082545335 2023-04-17 08:00:51,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/6abd5a44dfc74753bd53e709082b6595 2023-04-17 08:00:51,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/62b51e97cb204df0a785f5e8ded75714 2023-04-17 08:00:51,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits/704.seqid, newMaxSeqId=704, maxSeqId=4 2023-04-17 08:00:51,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72. 2023-04-17 08:00:51,356 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for f26affd8ee8c5b2a0826f19ce7699b72: 2023-04-17 08:00:51,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:51,357 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f26affd8ee8c5b2a0826f19ce7699b72, regionState=CLOSED 2023-04-17 08:00:51,358 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718451357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718451357"}]},"ts":"1681718451357"} 2023-04-17 08:00:51,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-17 08:00:51,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure f26affd8ee8c5b2a0826f19ce7699b72, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.9440 sec 2023-04-17 08:00:51,361 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-17 08:00:51,361 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f26affd8ee8c5b2a0826f19ce7699b72, UNASSIGN in 1.9480 sec 2023-04-17 08:00:51,362 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718451362"}]},"ts":"1681718451362"} 2023-04-17 08:00:51,363 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:00:51,394 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:00:51,397 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0040 sec 2023-04-17 08:00:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 08:00:51,505 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-17 08:00:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:00:51,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,511 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 08:00:51,513 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,521 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:51,525 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits] 2023-04-17 08:00:51,529 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/00246ecc3b534691832cecf19d2217bc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/00246ecc3b534691832cecf19d2217bc 2023-04-17 08:00:51,531 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d108abbedc214670a35aeae0f832b5f4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/d108abbedc214670a35aeae0f832b5f4 2023-04-17 08:00:51,533 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e6636bf17d444de39d111881adbb15c9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/A/e6636bf17d444de39d111881adbb15c9 2023-04-17 08:00:51,536 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/33758a27f2ee4e1aaee1e9fa01b7e588 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/33758a27f2ee4e1aaee1e9fa01b7e588 2023-04-17 08:00:51,537 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3a109ac01a2c4639b7f0f4c1996fd3ab to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/3a109ac01a2c4639b7f0f4c1996fd3ab 2023-04-17 08:00:51,539 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/700fb7af873f44768d2688727cfbebe3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/B/700fb7af873f44768d2688727cfbebe3 2023-04-17 08:00:51,543 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/30d81ee3580c4362ae304648a81333f7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/30d81ee3580c4362ae304648a81333f7 2023-04-17 08:00:51,545 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b39aeac1c7634c56b414ab053afd70c4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/b39aeac1c7634c56b414ab053afd70c4 2023-04-17 08:00:51,546 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/f1bdc48075e24beeb8069fb8064dcbbc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/C/f1bdc48075e24beeb8069fb8064dcbbc 2023-04-17 08:00:51,549 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits/704.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72/recovered.edits/704.seqid 2023-04-17 08:00:51,551 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/f26affd8ee8c5b2a0826f19ce7699b72 2023-04-17 08:00:51,551 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:00:51,551 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:00:51,552 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 08:00:51,557 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417074877402ae949d08316c320f24a303a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417074877402ae949d08316c320f24a303a 2023-04-17 08:00:51,558 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417107df9f6a25e4b8eb7866336f0648071 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417107df9f6a25e4b8eb7866336f0648071 2023-04-17 08:00:51,560 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711f2618f58d243f88ef9b1318e660d4b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711f2618f58d243f88ef9b1318e660d4b 2023-04-17 08:00:51,561 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171ce838096afb4972a6aee262c31c5abe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171ce838096afb4972a6aee262c31c5abe 2023-04-17 08:00:51,562 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f550c4387b6442ab4db0a78cbd44247 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f550c4387b6442ab4db0a78cbd44247 2023-04-17 08:00:51,563 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417232c74b538cf4b09ad0181a2fe749d26 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417232c74b538cf4b09ad0181a2fe749d26 2023-04-17 08:00:51,564 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417419ac54f216b4450b404d63c4742f556 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417419ac54f216b4450b404d63c4742f556 2023-04-17 08:00:51,565 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417534ed0c9134f4a23938cc10dbc815e4a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417534ed0c9134f4a23938cc10dbc815e4a 2023-04-17 08:00:51,566 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177d6dc575981c409b87c0bbd175b3b7d4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177d6dc575981c409b87c0bbd175b3b7d4 2023-04-17 08:00:51,566 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177e1e50dc8a694bd084c6f729d748341b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177e1e50dc8a694bd084c6f729d748341b 2023-04-17 08:00:51,567 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178178d046e5e84b639f07cc743f2263ee to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178178d046e5e84b639f07cc743f2263ee 2023-04-17 08:00:51,568 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178291736a0bfd4c08a7ccfd8186b68c92 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178291736a0bfd4c08a7ccfd8186b68c92 2023-04-17 08:00:51,569 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a222f172b0dd46b2b3b7f714121994d3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a222f172b0dd46b2b3b7f714121994d3 2023-04-17 08:00:51,570 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ba7978b8cd484b4b8939ce1434e0aa26 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ba7978b8cd484b4b8939ce1434e0aa26 2023-04-17 08:00:51,571 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c73181c0d9b84a9d94e3d20a55724d42 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c73181c0d9b84a9d94e3d20a55724d42 2023-04-17 08:00:51,572 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cb16edd824764e319809f1476fe6c887 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cb16edd824764e319809f1476fe6c887 2023-04-17 08:00:51,573 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d22391ac68c845fe969c2c93fde4d002 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d22391ac68c845fe969c2c93fde4d002 2023-04-17 08:00:51,574 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d918965fc5b54e7486e670598846dfef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d918965fc5b54e7486e670598846dfef 2023-04-17 08:00:51,574 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da93c3d890b145caa428437aa637f8c1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da93c3d890b145caa428437aa637f8c1 2023-04-17 08:00:51,575 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e66c6649c9c9405cb7958fd7d29ca459 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e66c6649c9c9405cb7958fd7d29ca459 2023-04-17 08:00:51,576 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f50120c0b9aa44a5bbfa80f16f2ce93c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f50120c0b9aa44a5bbfa80f16f2ce93c 2023-04-17 08:00:51,576 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:00:51,578 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,580 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:00:51,581 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:00:51,582 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,582 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:00:51,582 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718451582"}]},"ts":"9223372036854775807"} 2023-04-17 08:00:51,584 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:00:51,584 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f26affd8ee8c5b2a0826f19ce7699b72, NAME => 'TestAcidGuarantees,,1681718414803.f26affd8ee8c5b2a0826f19ce7699b72.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:00:51,584 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:00:51,584 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718451584"}]},"ts":"9223372036854775807"} 2023-04-17 08:00:51,585 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:00:51,595 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,596 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 88 msec 2023-04-17 08:00:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 08:00:51,614 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-17 08:00:51,631 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 235), OpenFileDescriptor=495 (was 582), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 363), ProcessCount=167 (was 167), AvailableMemoryMB=7707 (was 8023) 2023-04-17 08:00:51,643 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=495, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=167, AvailableMemoryMB=7707 2023-04-17 08:00:51,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:00:51,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:00:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 08:00:51,649 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 08:00:51,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-17 08:00:51,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 08:00:51,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:51,651 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:51,651 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 empty. 2023-04-17 08:00:51,652 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:51,652 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:00:51,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:52,061 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:00:52,065 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a8d6cbd896bae9327132746dceb281c8, NAME => 'TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 08:00:52,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:52,475 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:00:52,475 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a8d6cbd896bae9327132746dceb281c8, disabling compactions & flushes 2023-04-17 08:00:52,475 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,475 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,475 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. after waiting 0 ms 2023-04-17 08:00:52,475 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,476 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:52,481 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 08:00:52,483 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718452482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718452482"}]},"ts":"1681718452482"} 2023-04-17 08:00:52,484 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 08:00:52,485 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 08:00:52,486 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718452485"}]},"ts":"1681718452485"} 2023-04-17 08:00:52,487 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 08:00:52,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, ASSIGN}] 2023-04-17 08:00:52,532 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, ASSIGN 2023-04-17 08:00:52,534 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 08:00:52,686 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a8d6cbd896bae9327132746dceb281c8, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:52,687 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718452686"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718452686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718452686"}]},"ts":"1681718452686"} 2023-04-17 08:00:52,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:00:52,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:52,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8d6cbd896bae9327132746dceb281c8, NAME => 'TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:00:52,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:00:52,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,856 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,859 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A 2023-04-17 08:00:52,859 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A 2023-04-17 08:00:52,859 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:52,859 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8d6cbd896bae9327132746dceb281c8 columnFamilyName A 2023-04-17 08:00:52,860 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(310): Store=a8d6cbd896bae9327132746dceb281c8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:52,860 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,862 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B 2023-04-17 08:00:52,862 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B 2023-04-17 08:00:52,863 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:52,863 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8d6cbd896bae9327132746dceb281c8 columnFamilyName B 2023-04-17 08:00:52,863 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(310): Store=a8d6cbd896bae9327132746dceb281c8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:52,864 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,865 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C 2023-04-17 08:00:52,865 DEBUG [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C 2023-04-17 08:00:52,866 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:00:52,866 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8d6cbd896bae9327132746dceb281c8 columnFamilyName C 2023-04-17 08:00:52,866 INFO [StoreOpener-a8d6cbd896bae9327132746dceb281c8-1] regionserver.HStore(310): Store=a8d6cbd896bae9327132746dceb281c8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:00:52,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:00:52,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:52,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 08:00:52,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened a8d6cbd896bae9327132746dceb281c8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75054082, jitterRate=0.11839297413825989}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:00:52,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:52,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., pid=30, masterSystemTime=1681718452844 2023-04-17 08:00:52,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:52,874 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a8d6cbd896bae9327132746dceb281c8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:52,874 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718452874"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718452874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718452874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718452874"}]},"ts":"1681718452874"} 2023-04-17 08:00:52,876 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-17 08:00:52,876 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 in 185 msec 2023-04-17 08:00:52,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-17 08:00:52,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, ASSIGN in 348 msec 2023-04-17 08:00:52,878 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 08:00:52,878 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718452878"}]},"ts":"1681718452878"} 2023-04-17 08:00:52,879 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 08:00:52,912 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 08:00:52,914 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2660 sec 2023-04-17 08:00:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 08:00:53,758 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-17 08:00:53,760 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,813 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,815 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,818 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,820 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 08:00:53,824 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:43344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 08:00:53,826 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,840 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,840 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,854 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,855 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5a8a575d to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,871 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e3efec6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,872 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x6347f2e6 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,887 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b7f716c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,888 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x36523181 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,904 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ade356f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,905 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,920 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,920 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,931 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,931 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5d22a55e to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,945 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@993cc7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,945 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x2179ead9 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,957 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69b9b3fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,957 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x7897b06a to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:00:53,970 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1656b283, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:00:53,978 DEBUG [hconnection-0x7284a57d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,979 DEBUG [hconnection-0x271dfba9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,979 DEBUG [hconnection-0x50fabd03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:53,980 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,985 DEBUG [hconnection-0x27c81d4b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,985 DEBUG [hconnection-0xd5f5fe8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,989 DEBUG [hconnection-0x57ca12c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,991 DEBUG [hconnection-0x25549253-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,991 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,991 DEBUG [hconnection-0x4843fe98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,991 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,992 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,991 DEBUG [hconnection-0x5802b13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,991 DEBUG [hconnection-0x3a5302b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:00:53,992 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,991 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,995 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:53,997 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:54,000 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:54,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:54,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:54,000 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:54,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,001 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:54,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:54,003 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,003 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,004 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:00:54,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:54,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:54,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:54,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:54,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,015 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:54,019 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:54,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:54,020 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:54,020 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:54,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:54,020 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:54,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:54,022 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:54,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:54,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:54,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:54,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:54,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:54,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:54,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:54,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:54,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:54,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33afb09e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:54,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718454028, max timestamp=1681718454034Num uniques 859; 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718454028, max timestamp=1681718454034Num uniques 859; 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718454028, max timestamp=1681718454034Num uniques 859; 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:54,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:00:54,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:54,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:54,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:54,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:54,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718514045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718514047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718514072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718514072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718514072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,109 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:54,109 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,109 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,110 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:54,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:54,113 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:54,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:54,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:54,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:54,113 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:54,183 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,183 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,183 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,185 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718514184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718514173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718514185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718514189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:54,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:54,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,198 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,198 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,198 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:54,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718514185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,298 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:54,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:54,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:54,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:54,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:54,301 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:54,301 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,301 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:54,301 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:54,302 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,302 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,311 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,311 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,311 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,311 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:54,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:54,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:54,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:54,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:54,320 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:54,320 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:54,320 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:54,320 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:54,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:54,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:54,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20efa0e6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:54,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:54,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:54,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:54,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:54,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:54,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:54,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:54,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:54,350 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:54,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:54,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:54,350 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:54,350 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:54,361 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,361 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,361 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:54,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,373 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,373 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:54,373 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:54,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718514389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718514390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718514390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718514397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718514399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/7f89d0aa9f924a6bbbd824d783604bf4 2023-04-17 08:00:54,473 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:54,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:54,476 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:54,476 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,476 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:54,476 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:54,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,518 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,518 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,519 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,519 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:54,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:54,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:54,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:54,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:54,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:54,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:54,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:54,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:54,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:54,547 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:54,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573b282f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:54,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:54,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:54,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:54,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:54,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:54,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:54,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:54,564 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:54,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:54,565 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:54,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:54,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:54,566 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:54,566 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:54,572 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,572 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,572 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:54,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:54,581 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,581 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:54,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,581 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,581 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,681 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:54,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:54,684 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:54,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:54,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:54,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718514692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718514692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718514693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,697 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:54,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:54,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:54,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:54,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:54,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:54,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:54,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:54,699 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:54,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718514700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718514702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:54,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4e9eeb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:54,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:54,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:54,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:54,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:54,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:54,722 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:54,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:54,724 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:54,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:54,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:54,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:54,724 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:54,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,731 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:54,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,739 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,739 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:54,739 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,839 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:54,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:54,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:54,842 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:54,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:54,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:54,842 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,843 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,852 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,852 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,853 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:54,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:54,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:54,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:54,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:54,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:54,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:54,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:54,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:54,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73abe29c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:54,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/5744695b8c4847948df6879d3ed3cdf3 2023-04-17 08:00:54,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:54,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:54,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:54,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:54,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:54,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,881 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:54,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:54,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:54,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:54,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:54,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:54,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:54,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:54,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:54,897 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:54,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:54,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,897 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:54,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:54,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:54,898 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:54,998 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:54,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,002 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,011 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,011 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,011 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,011 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:55,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:55,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e9e949f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,052 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,053 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,053 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,061 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,063 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,063 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:55,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,081 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,081 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,181 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:55,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,186 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,194 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,194 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:55,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718515196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:55,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718515198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,200 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,200 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,201 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:55,201 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,201 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718515199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718515206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,211 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,211 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,211 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,211 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718515209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20eaab33[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,228 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,228 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,228 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,228 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,236 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,238 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,238 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,249 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:55,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,256 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,256 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,256 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/4963d2339deb4a05b7877c7d7d95382d 2023-04-17 08:00:55,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/7f89d0aa9f924a6bbbd824d783604bf4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4 2023-04-17 08:00:55,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 08:00:55,288 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/5744695b8c4847948df6879d3ed3cdf3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3 2023-04-17 08:00:55,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 08:00:55,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/4963d2339deb4a05b7877c7d7d95382d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d 2023-04-17 08:00:55,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d, entries=150, sequenceid=13, filesize=11.6 K 2023-04-17 08:00:55,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for a8d6cbd896bae9327132746dceb281c8 in 1281ms, sequenceid=13, compaction requested=false 2023-04-17 08:00:55,302 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 08:00:55,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:55,356 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:55,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,359 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:55,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,371 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,371 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:55,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:55,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,380 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/6a4f868e94f24655bd8377f1a02e4e88 2023-04-17 08:00:55,390 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/6970a6d4fee44d2db6c80a753ccbde9f 2023-04-17 08:00:55,403 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/99d7b13793794f36a8b9a9ff23e0a0ec 2023-04-17 08:00:55,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/6a4f868e94f24655bd8377f1a02e4e88 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88 2023-04-17 08:00:55,410 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:00:55,411 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/6970a6d4fee44d2db6c80a753ccbde9f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f 2023-04-17 08:00:55,426 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:00:55,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/99d7b13793794f36a8b9a9ff23e0a0ec as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec 2023-04-17 08:00:55,430 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:00:55,431 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for a8d6cbd896bae9327132746dceb281c8 in 60ms, sequenceid=48, compaction requested=false 2023-04-17 08:00:55,433 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:55,437 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,447 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,447 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699ac003[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,466 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,466 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,466 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,466 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,466 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,481 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,481 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,581 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:55,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,585 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,594 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,594 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,594 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:55,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,599 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,599 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,599 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:55,599 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450970ae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,614 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,650 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,650 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,664 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:55,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,664 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,664 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,764 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:55,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:55,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,767 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,777 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:55,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:55,780 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,789 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f692efc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,805 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,805 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,805 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,806 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,807 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,808 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,818 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,827 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,827 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,827 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:55,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,828 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,828 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,928 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:55,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:55,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:55,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:55,931 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:55,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:55,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:55,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,939 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:55,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:55,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:55,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:55,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:55,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:55,941 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,941 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:55,941 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:55,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:55,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2fad11[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:55,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:55,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:55,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:55,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:55,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:55,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:55,964 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:55,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:55,966 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:55,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:55,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:55,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:55,966 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:55,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:55,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:55,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,974 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:55,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:55,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:55,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:55,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:55,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:55,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:55,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:55,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:55,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:55,981 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:55,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:55,981 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,081 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,084 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,094 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,095 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,095 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,095 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,096 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,096 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,096 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:56,096 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550af28f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,114 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,124 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,124 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,186 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,186 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,186 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,186 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:56,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:56,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718516225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718516226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718516226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718456223, max timestamp=1681718456227Num uniques 859; 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718456223, max timestamp=1681718456227Num uniques 859; 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718456223, max timestamp=1681718456227Num uniques 859; 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:56,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:00:56,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718516257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718516262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,287 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,289 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,298 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,299 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,299 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,299 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:56,299 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,299 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,305 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b887480[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,314 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,322 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,322 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,322 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,322 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,324 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,324 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,324 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,324 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,324 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718516329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718516329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718516329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,339 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,339 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718516363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718516366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,439 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,442 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,442 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,443 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,443 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,452 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,454 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,454 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,454 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:56,454 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc0afc3[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,481 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,483 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,483 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,483 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,491 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:56,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,498 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,498 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718516531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718516531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718516531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718516567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718516569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,598 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,600 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,611 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,611 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,611 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,611 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:56,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e12fd70e7aaf402b870969912e2e85ad 2023-04-17 08:00:56,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,623 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a10cc4f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,626 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,626 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,636 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,636 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,636 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,636 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,636 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,636 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,644 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,644 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,644 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,644 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,646 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,646 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,655 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,657 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,664 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,664 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,664 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,764 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,767 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,777 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:56,779 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,789 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,789 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188ccb08[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,805 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,805 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,805 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,805 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,807 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,807 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,814 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,814 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:56,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,822 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,823 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,823 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,823 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718516833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718516835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718516835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718516868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718516873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,889 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 08:00:56,923 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:56,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:56,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:56,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:56,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:56,925 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:56,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:56,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:56,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,926 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,936 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,936 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,936 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,936 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:56,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:56,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:56,937 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,937 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:56,937 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:56,937 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:56,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:56,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:56,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:56,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b880ed[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:56,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:56,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:56,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:56,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:56,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:56,964 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,964 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:56,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:56,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:56,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:56,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:56,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:56,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:56,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:56,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:56,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:56,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:56,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:56,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:56,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:56,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:56,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:56,981 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:56,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:56,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:56,981 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:56,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:56,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:56,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:57,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/df57b4709be84429b69a96a66f59471e 2023-04-17 08:00:57,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/a0f668eb3fae4deebae17fd6bb2ee20d 2023-04-17 08:00:57,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e12fd70e7aaf402b870969912e2e85ad as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad 2023-04-17 08:00:57,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 08:00:57,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/df57b4709be84429b69a96a66f59471e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e 2023-04-17 08:00:57,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 08:00:57,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/a0f668eb3fae4deebae17fd6bb2ee20d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d 2023-04-17 08:00:57,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 08:00:57,063 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a8d6cbd896bae9327132746dceb281c8 in 855ms, sequenceid=62, compaction requested=true 2023-04-17 08:00:57,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:57,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:57,063 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:57,064 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:57,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:57,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:57,064 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:57,064 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:57,064 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:00:57,064 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:00:57,064 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,064 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,065 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=34.8 K 2023-04-17 08:00:57,065 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=34.8 K 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5744695b8c4847948df6879d3ed3cdf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718454013 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7f89d0aa9f924a6bbbd824d783604bf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718454013 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6970a6d4fee44d2db6c80a753ccbde9f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718454046 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6a4f868e94f24655bd8377f1a02e4e88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718454046 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting df57b4709be84429b69a96a66f59471e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:57,065 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e12fd70e7aaf402b870969912e2e85ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:57,073 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#206 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:57,073 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:57,081 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:57,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:57,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:57,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:57,097 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:57,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:57,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:57,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,104 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d41cb64abac94c6aa7a3866a72bf2cd0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41cb64abac94c6aa7a3866a72bf2cd0 2023-04-17 08:00:57,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:57,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:57,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:57,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:57,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:57,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:57,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:57,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:57,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:57,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:57,107 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,107 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:57,108 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:57,108 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:00:57,112 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into d41cb64abac94c6aa7a3866a72bf2cd0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:57,112 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:57,112 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=13, startTime=1681718457063; duration=0sec 2023-04-17 08:00:57,112 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:57,113 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:57,116 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:57,116 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:00:57,116 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,116 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=34.8 K 2023-04-17 08:00:57,116 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/f9ca425ee1924ce08ceb3b636901474c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/f9ca425ee1924ce08ceb3b636901474c 2023-04-17 08:00:57,116 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4963d2339deb4a05b7877c7d7d95382d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681718454013 2023-04-17 08:00:57,117 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 99d7b13793794f36a8b9a9ff23e0a0ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718454046 2023-04-17 08:00:57,117 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a0f668eb3fae4deebae17fd6bb2ee20d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:57,121 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into f9ca425ee1924ce08ceb3b636901474c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:57,121 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:57,121 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=13, startTime=1681718457063; duration=0sec 2023-04-17 08:00:57,121 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:57,128 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#209 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:57,135 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/4119682511a2427495c4dfa261bbc637 2023-04-17 08:00:57,168 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/26570ec414ad4eeaafbe28745632586c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/26570ec414ad4eeaafbe28745632586c 2023-04-17 08:00:57,172 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 26570ec414ad4eeaafbe28745632586c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:57,172 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:57,172 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=13, startTime=1681718457064; duration=0sec 2023-04-17 08:00:57,172 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:57,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:57,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:57,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718517344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718517344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718517352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718517371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718517378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718517446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718517447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718517455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,570 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/a7ee43afbd294053977e38db699f8584 2023-04-17 08:00:57,578 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/ab468f707bfd4282b6847eaddf2a3c78 2023-04-17 08:00:57,582 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/4119682511a2427495c4dfa261bbc637 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637 2023-04-17 08:00:57,601 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637, entries=150, sequenceid=99, filesize=11.6 K 2023-04-17 08:00:57,602 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/a7ee43afbd294053977e38db699f8584 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584 2023-04-17 08:00:57,607 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584, entries=150, sequenceid=99, filesize=11.6 K 2023-04-17 08:00:57,608 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/ab468f707bfd4282b6847eaddf2a3c78 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78 2023-04-17 08:00:57,616 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78, entries=150, sequenceid=99, filesize=11.6 K 2023-04-17 08:00:57,620 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a8d6cbd896bae9327132746dceb281c8 in 513ms, sequenceid=99, compaction requested=false 2023-04-17 08:00:57,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:57,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:57,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:57,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:57,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:57,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:57,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:57,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:57,652 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,652 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,653 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,653 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:57,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:57,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@80e998e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:57,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/91271b071d8847ee85465dc0e0eba9d9 2023-04-17 08:00:57,664 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:57,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:57,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:57,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:57,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:57,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:57,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:57,675 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:57,675 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:57,675 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:57,675 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:57,675 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718517673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718517674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718517675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:57,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:57,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:57,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,712 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:57,712 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,712 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:57,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:57,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718517777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718517777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718517778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,813 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:57,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:57,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:57,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:57,815 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:57,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:57,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:57,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:57,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:57,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:57,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:57,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:57,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:57,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:57,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:57,823 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:57,823 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,823 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:57,824 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:57,824 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:57,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:57,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:57,824 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:57,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,831 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:57,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14604e74[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:57,833 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:57,833 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,839 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:57,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:57,839 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,839 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:57,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:57,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:57,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:57,847 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:57,849 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:57,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:57,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:57,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:57,849 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:57,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,856 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,857 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:57,864 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:57,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:57,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:57,864 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:57,864 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:57,964 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:57,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:57,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:57,967 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:57,967 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,967 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:57,967 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:57,967 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,967 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,977 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:57,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:57,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:57,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:57,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:57,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:57,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:57,979 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,979 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:57,979 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:57,979 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:57,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718517980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718517980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:57,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718517981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,989 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:57,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:57,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:57,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4a037b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:57,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:57,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:57,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:57,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:57,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:57,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:57,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:58,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:58,005 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:58,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:58,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:58,007 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:58,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:58,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:58,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:58,007 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:58,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:58,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:58,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:58,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:58,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:58,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:58,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:58,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:58,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:58,022 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:58,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:58,023 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:58,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:58,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:58,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/b655515fb658452195618c406b4d5cb0 2023-04-17 08:00:58,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/e874854a9569433885db48b58a9821d9 2023-04-17 08:00:58,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/91271b071d8847ee85465dc0e0eba9d9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9 2023-04-17 08:00:58,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9, entries=250, sequenceid=114, filesize=16.2 K 2023-04-17 08:00:58,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/b655515fb658452195618c406b4d5cb0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0 2023-04-17 08:00:58,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0, entries=150, sequenceid=114, filesize=11.6 K 2023-04-17 08:00:58,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/e874854a9569433885db48b58a9821d9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9 2023-04-17 08:00:58,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9, entries=150, sequenceid=114, filesize=11.6 K 2023-04-17 08:00:58,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a8d6cbd896bae9327132746dceb281c8 in 455ms, sequenceid=114, compaction requested=true 2023-04-17 08:00:58,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:58,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:58,108 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:58,108 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:58,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:58,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:58,109 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:58,109 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:00:58,109 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:58,109 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:58,109 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:00:58,109 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:58,109 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/f9ca425ee1924ce08ceb3b636901474c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=39.5 K 2023-04-17 08:00:58,109 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41cb64abac94c6aa7a3866a72bf2cd0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=34.9 K 2023-04-17 08:00:58,111 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f9ca425ee1924ce08ceb3b636901474c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:58,111 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d41cb64abac94c6aa7a3866a72bf2cd0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:58,111 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4119682511a2427495c4dfa261bbc637, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681718456233 2023-04-17 08:00:58,111 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a7ee43afbd294053977e38db699f8584, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681718456233 2023-04-17 08:00:58,111 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 91271b071d8847ee85465dc0e0eba9d9, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:00:58,112 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b655515fb658452195618c406b4d5cb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:00:58,120 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#215 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:58,122 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:58,123 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:58,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:58,128 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:58,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:58,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:58,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:58,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:58,136 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:58,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:58,136 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:58,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:58,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:58,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:58,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:58,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:58,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:58,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:58,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:58,138 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:58,138 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:58,139 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:58,143 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/4766ba7db1f84a81ac72e279e05294f3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4766ba7db1f84a81ac72e279e05294f3 2023-04-17 08:00:58,150 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into 4766ba7db1f84a81ac72e279e05294f3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:58,150 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:58,150 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=13, startTime=1681718458108; duration=0sec 2023-04-17 08:00:58,150 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:00:58,150 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:00:58,157 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:00:58,160 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:00:58,160 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:58,160 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/26570ec414ad4eeaafbe28745632586c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=34.9 K 2023-04-17 08:00:58,169 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 26570ec414ad4eeaafbe28745632586c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681718456203 2023-04-17 08:00:58,169 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/3b920ac473c94e51919e0ebcc9e7c89c 2023-04-17 08:00:58,169 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ab468f707bfd4282b6847eaddf2a3c78, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681718456233 2023-04-17 08:00:58,170 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e874854a9569433885db48b58a9821d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:00:58,181 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:00:58,221 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/8dac69cfd26f4a9a8ee8c106290b0d23 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8dac69cfd26f4a9a8ee8c106290b0d23 2023-04-17 08:00:58,231 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 8dac69cfd26f4a9a8ee8c106290b0d23(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:58,231 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:58,231 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=13, startTime=1681718458108; duration=0sec 2023-04-17 08:00:58,231 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:58,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:58,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718518289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718518289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718518290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718518376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718518387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718518391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718518392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718518392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,562 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/4bb5a4aca99f4c88b43295de959a4643 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4bb5a4aca99f4c88b43295de959a4643 2023-04-17 08:00:58,566 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into 4bb5a4aca99f4c88b43295de959a4643(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:00:58,566 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:58,566 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=13, startTime=1681718458108; duration=0sec 2023-04-17 08:00:58,566 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:00:58,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718518593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718518595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718518595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,631 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/176be35b894d4354b07cc3b263d38440 2023-04-17 08:00:58,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718518897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718518898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:58,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718518899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,059 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/8398b6c66a314dd09f7f4de1c2836fd9 2023-04-17 08:00:59,066 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/3b920ac473c94e51919e0ebcc9e7c89c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c 2023-04-17 08:00:59,072 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 08:00:59,073 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/176be35b894d4354b07cc3b263d38440 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440 2023-04-17 08:00:59,093 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 08:00:59,094 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/8398b6c66a314dd09f7f4de1c2836fd9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9 2023-04-17 08:00:59,110 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 08:00:59,110 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a8d6cbd896bae9327132746dceb281c8 in 972ms, sequenceid=134, compaction requested=false 2023-04-17 08:00:59,111 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:00:59,111 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:59,111 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:00:59,111 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:00:59,111 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:00:59,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:00:59,144 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,144 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:59,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:59,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:00:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:59,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:00:59,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eb15937[Count = 0] remaining members to acquire global barrier 2023-04-17 08:00:59,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:00:59,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,155 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,156 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,156 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:00:59,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:00:59,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:00:59,156 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:00:59,164 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:00:59,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,164 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:59,164 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:00:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:00:59,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:00:59,168 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:00:59,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:00:59,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:00:59,168 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:00:59,168 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:00:59,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:59,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:00:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:00:59,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:59,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:00:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:00:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:00:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:00:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:59,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:00:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:00:59,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:00:59,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:00:59,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:00:59,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:00:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:00:59,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:00:59,263 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:00:59,263 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:00:59,363 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:00:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:00:59,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:00:59,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:00:59,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:00:59,365 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:00:59,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:59,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:00:59,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:00:59,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,366 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:59,377 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:00:59,377 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:00:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:00:59,378 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,378 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:00:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:59,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:00:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:00:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:00:59,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:00:59,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:00:59,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:00:59,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:00:59,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:00:59,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:00:59,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:00:59,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:00:59,379 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:00:59,379 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:59,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:00:59,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:00:59,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:00:59,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:00:59,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718519418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718519419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718519420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718519521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718519521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718519522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718519723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718519725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:00:59,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718519725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:00:59,786 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/fe3fb90b533e407c9c8bc57d146e7029 2023-04-17 08:01:00,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718520027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718520027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718520029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,207 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/28ae1d8db410420ba47d17ce847c7e22 2023-04-17 08:01:00,215 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/6de3d060cdcc4ea39eeea85bae51bd16 2023-04-17 08:01:00,219 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/fe3fb90b533e407c9c8bc57d146e7029 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029 2023-04-17 08:01:00,223 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029, entries=150, sequenceid=149, filesize=11.8 K 2023-04-17 08:01:00,223 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/28ae1d8db410420ba47d17ce847c7e22 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22 2023-04-17 08:01:00,226 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22, entries=150, sequenceid=149, filesize=11.8 K 2023-04-17 08:01:00,227 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/6de3d060cdcc4ea39eeea85bae51bd16 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16 2023-04-17 08:01:00,230 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16, entries=150, sequenceid=149, filesize=11.8 K 2023-04-17 08:01:00,231 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a8d6cbd896bae9327132746dceb281c8 in 852ms, sequenceid=149, compaction requested=true 2023-04-17 08:01:00,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:00,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:00,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:00,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:00,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:00,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:00,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@528f7786[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:00,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,302 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:00,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:00,302 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:00,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,303 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:00,311 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:00,311 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,311 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,311 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:00,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:00,313 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:00,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:00,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:00,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:00,313 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:00,322 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,322 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,322 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,322 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:00,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:00,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,331 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,331 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:00,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:00,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:00,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:00,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718520397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718520398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,431 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:00,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:00,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:00,434 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:00,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:00,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:00,435 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,435 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,444 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,444 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,444 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,444 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:00,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:00,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:00,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:00,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:00,446 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,446 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:00,446 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:00,446 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:00,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:00,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:00,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@440a904d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:00,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:00,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:00,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:00,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:00,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:00,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:00,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:00,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:00,474 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:00,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:00,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:00,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:00,474 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:00,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:00,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:00,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:00,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,489 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:00,489 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:00,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718520500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718520501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718520531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718520532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718520534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,589 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:00,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:00,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:00,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:00,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:00,608 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:00,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:00,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:00,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,608 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,614 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:00,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:00,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:00,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:00,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:00,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:00,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:00,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:00,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:00,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:00,620 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:00,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:00,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:00,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e61126c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:00,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:00,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,639 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:00,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:00,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:00,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:00,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:00,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:00,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:00,651 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:00,651 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:00,651 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:00,651 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:00,651 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:00,661 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,661 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,661 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,663 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:00,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:00,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,673 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,673 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:00,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,673 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718520702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718520704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,774 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:00,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:00,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:00,776 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:00,776 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,776 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:00,776 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:00,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,786 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,786 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,786 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,786 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:00,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:00,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/a3d77e1460cb435590c823eb8d3c9222 2023-04-17 08:01:00,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:00,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:00,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:00,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:00,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:00,787 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,787 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:00,787 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:00,787 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:00,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:00,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:00,788 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:00,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,798 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:00,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abd8476[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:00,799 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:00,799 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,847 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:00,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:00,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:00,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:00,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:00,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,864 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:00,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:00,866 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:00,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:00,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:00,866 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:00,866 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:00,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:00,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:00,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:00,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:00,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:00,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,881 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:00,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:00,881 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:00,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:00,881 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,981 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:00,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:00,984 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:00,984 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,984 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:00,984 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:00,984 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,984 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,994 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:00,994 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:00,994 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,994 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:00,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:00,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:00,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:00,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:00,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:00,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:00,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:00,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:00,995 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,995 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:00,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:00,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:00,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:00,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:00,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:00,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:00,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:01,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718521004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,005 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:01,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7327ebe0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:01,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:01,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718521010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,014 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:01,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:01,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:01,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:01,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:01,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:01,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:01,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:01,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:01,029 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:01,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:01,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:01,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:01,029 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:01,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:01,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,041 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:01,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:01,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:01,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:01,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:01,048 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:01,048 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:01,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:01,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,048 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:01,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:01,148 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:01,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:01,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:01,153 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:01,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:01,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:01,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:01,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,161 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:01,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:01,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:01,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:01,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:01,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:01,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:01,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:01,172 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:01,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63bf9531[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:01,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:01,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:01,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:01,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:01,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:01,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:01,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:01,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:01,191 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:01,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:01,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:01,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:01,191 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:01,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/b66ad99d33ae442c88e8de71ecb861e5 2023-04-17 08:01:01,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:01,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:01,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:01,201 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f9ff20ffd8f64b5bb864ea5827f4b49f 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:01,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:01,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:01,214 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:01,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:01,214 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:01,215 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:01,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/a3d77e1460cb435590c823eb8d3c9222 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222 2023-04-17 08:01:01,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222, entries=200, sequenceid=170, filesize=14.1 K 2023-04-17 08:01:01,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/b66ad99d33ae442c88e8de71ecb861e5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5 2023-04-17 08:01:01,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 08:01:01,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f9ff20ffd8f64b5bb864ea5827f4b49f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f 2023-04-17 08:01:01,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 08:01:01,233 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for a8d6cbd896bae9327132746dceb281c8 in 852ms, sequenceid=170, compaction requested=true 2023-04-17 08:01:01,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:01,233 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:01,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:01,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:01,233 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:01,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:01,236 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50590 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:01,236 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:01,236 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:01,236 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:01,237 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,237 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4bb5a4aca99f4c88b43295de959a4643, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=49.4 K 2023-04-17 08:01:01,237 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4766ba7db1f84a81ac72e279e05294f3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=47.1 K 2023-04-17 08:01:01,237 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4bb5a4aca99f4c88b43295de959a4643, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:01:01,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4766ba7db1f84a81ac72e279e05294f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:01:01,237 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 3b920ac473c94e51919e0ebcc9e7c89c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718457671 2023-04-17 08:01:01,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 176be35b894d4354b07cc3b263d38440, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718457671 2023-04-17 08:01:01,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting fe3fb90b533e407c9c8bc57d146e7029, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681718458287 2023-04-17 08:01:01,238 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 28ae1d8db410420ba47d17ce847c7e22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681718458287 2023-04-17 08:01:01,238 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b66ad99d33ae442c88e8de71ecb861e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459418 2023-04-17 08:01:01,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a3d77e1460cb435590c823eb8d3c9222, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459413 2023-04-17 08:01:01,245 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:01,248 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:01,255 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/647ec4c36738436ab78bb200c1fa949b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/647ec4c36738436ab78bb200c1fa949b 2023-04-17 08:01:01,265 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/dc2fbaebf84f44679f2f8c6fb4965f9d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/dc2fbaebf84f44679f2f8c6fb4965f9d 2023-04-17 08:01:01,266 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into 647ec4c36738436ab78bb200c1fa949b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:01,267 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:01,267 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=12, startTime=1681718461233; duration=0sec 2023-04-17 08:01:01,267 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:01,267 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:01,268 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:01,268 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:01,268 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,268 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8dac69cfd26f4a9a8ee8c106290b0d23, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=47.1 K 2023-04-17 08:01:01,268 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8dac69cfd26f4a9a8ee8c106290b0d23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681718457340 2023-04-17 08:01:01,269 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8398b6c66a314dd09f7f4de1c2836fd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718457671 2023-04-17 08:01:01,269 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6de3d060cdcc4ea39eeea85bae51bd16, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681718458287 2023-04-17 08:01:01,269 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f9ff20ffd8f64b5bb864ea5827f4b49f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459418 2023-04-17 08:01:01,277 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:01,278 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into dc2fbaebf84f44679f2f8c6fb4965f9d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:01,278 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:01,278 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=12, startTime=1681718461233; duration=0sec 2023-04-17 08:01:01,278 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:01,312 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/38f0276f61b44858bb61257d5141f4bd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/38f0276f61b44858bb61257d5141f4bd 2023-04-17 08:01:01,315 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:01,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:01,319 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 38f0276f61b44858bb61257d5141f4bd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:01,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:01,319 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=12, startTime=1681718461233; duration=0sec 2023-04-17 08:01:01,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:01,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:01,320 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:01,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:01,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:01,320 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:01,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:01,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:01,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:01,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:01,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:01,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:01,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:01,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:01,332 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:01,336 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:01,336 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:01,336 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:01,336 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:01,337 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:01,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:01,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:01,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718521527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718461519, max timestamp=1681718461528Num uniques 859; 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718461519, max timestamp=1681718461528Num uniques 859; 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718461519, max timestamp=1681718461528Num uniques 859; 2023-04-17 08:01:01,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:01,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718521566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718521567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718521567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718521568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718521631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718521670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718521671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718521672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718521674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,755 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/85d54683fdca4f87a6fc793b39d16ec1 2023-04-17 08:01:01,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718521832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718521874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718521874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718521876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:01,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:01,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718521879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718522134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718522178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718522179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718522179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718522187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,200 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/5e911f833d314cd884dcd4efb2421f7f 2023-04-17 08:01:02,251 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f5343d9e9c3e4081996f79212fe57225 2023-04-17 08:01:02,258 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/85d54683fdca4f87a6fc793b39d16ec1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1 2023-04-17 08:01:02,266 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1, entries=150, sequenceid=184, filesize=11.8 K 2023-04-17 08:01:02,267 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/5e911f833d314cd884dcd4efb2421f7f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f 2023-04-17 08:01:02,279 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f, entries=150, sequenceid=184, filesize=11.8 K 2023-04-17 08:01:02,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f5343d9e9c3e4081996f79212fe57225 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225 2023-04-17 08:01:02,285 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225, entries=150, sequenceid=184, filesize=11.8 K 2023-04-17 08:01:02,296 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a8d6cbd896bae9327132746dceb281c8 in 960ms, sequenceid=184, compaction requested=false 2023-04-17 08:01:02,296 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:02,296 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:02,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:02,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:02,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:02,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:02,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:02,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:02,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:02,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@153b1958[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:02,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:02,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:02,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:02,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:02,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:02,368 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,368 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:02,368 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:02,368 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:02,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:02,371 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:02,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:02,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:02,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:02,371 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:02,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:02,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:02,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:02,383 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:02,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:02,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:02,394 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:02,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:02,394 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:02,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:02,495 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:02,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:02,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:02,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:02,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:02,497 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:02,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:02,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:02,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:02,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:02,505 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:02,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:02,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:02,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:02,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:02,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:02,507 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:02,507 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:02,514 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/9348febbcbbd4be2ae627b8d9db51459 2023-04-17 08:01:02,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:02,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:02,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718522666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718522685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718522685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718522686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718522694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718522769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:02,942 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/8319c9fd4a75453b93437ef923427c7e 2023-04-17 08:01:02,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:02,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718522973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718523275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,351 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/2722558cbe7243818b213ade6b5975ae 2023-04-17 08:01:03,355 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/9348febbcbbd4be2ae627b8d9db51459 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459 2023-04-17 08:01:03,358 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459, entries=150, sequenceid=220, filesize=11.8 K 2023-04-17 08:01:03,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/8319c9fd4a75453b93437ef923427c7e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e 2023-04-17 08:01:03,362 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e, entries=150, sequenceid=220, filesize=11.8 K 2023-04-17 08:01:03,363 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/2722558cbe7243818b213ade6b5975ae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae 2023-04-17 08:01:03,366 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae, entries=150, sequenceid=220, filesize=11.8 K 2023-04-17 08:01:03,367 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a8d6cbd896bae9327132746dceb281c8 in 860ms, sequenceid=220, compaction requested=true 2023-04-17 08:01:03,367 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:03,367 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:03,367 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:03,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:03,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:03,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2393b1bd[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:03,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:03,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,389 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:03,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:03,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:03,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:03,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:03,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:03,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:03,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:03,399 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:03,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:03,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:03,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:03,399 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:03,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:03,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:03,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:03,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:03,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:03,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:03,414 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:03,414 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:03,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,514 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:03,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:03,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:03,517 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:03,517 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:03,517 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:03,517 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:03,518 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,518 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:03,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,527 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:03,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:03,527 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,528 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:03,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:03,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:03,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:03,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:03,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:03,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:03,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:03,529 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:03,529 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:03,529 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 08:01:03,529 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:03,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:03,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:03,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:03,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:03,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:03,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:03,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718463710, max timestamp=1681718463717Num uniques 859; 2023-04-17 08:01:03,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:03,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718463710, max timestamp=1681718463717Num uniques 859; 2023-04-17 08:01:03,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:03,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718463710, max timestamp=1681718463717Num uniques 859; 2023-04-17 08:01:03,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:01:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718523739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718523740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718523740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718523743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718523783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718523846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718523846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718523846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718523848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:03,939 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e90996d32b7a4c67a37c110ac8c97ed8 2023-04-17 08:01:04,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718524049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718524049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718524049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718524050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,348 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/394e5880c88a44998569a6139b02f4a8 2023-04-17 08:01:04,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718524351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718524352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718524353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718524354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,356 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/972716e12a2c474aa31249a4bd68399a 2023-04-17 08:01:04,360 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e90996d32b7a4c67a37c110ac8c97ed8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8 2023-04-17 08:01:04,363 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8, entries=150, sequenceid=231, filesize=11.8 K 2023-04-17 08:01:04,364 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/394e5880c88a44998569a6139b02f4a8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8 2023-04-17 08:01:04,367 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8, entries=150, sequenceid=231, filesize=11.8 K 2023-04-17 08:01:04,368 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/972716e12a2c474aa31249a4bd68399a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a 2023-04-17 08:01:04,371 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a, entries=150, sequenceid=231, filesize=11.8 K 2023-04-17 08:01:04,372 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a8d6cbd896bae9327132746dceb281c8 in 843ms, sequenceid=231, compaction requested=true 2023-04-17 08:01:04,372 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:04,372 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:04,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:04,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:04,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:04,373 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:04,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:04,416 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:04,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:04,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa726c4[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:04,417 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:04,417 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,435 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,436 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,436 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,436 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:04,436 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:04,436 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:04,452 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:04,452 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,452 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:04,452 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:04,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:04,454 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:04,454 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:04,454 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:04,454 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:04,454 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:04,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:04,464 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:04,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:04,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:04,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:04,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:04,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:04,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:04,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:04,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:04,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:04,472 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,472 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:04,473 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:04,573 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:04,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:04,575 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:04,575 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:04,575 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:04,575 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:04,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:04,585 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,585 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:04,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:04,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:04,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:04,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:04,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:04,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:04,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:04,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:04,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:04,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:04,587 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:04,587 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:04,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:04,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:04,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718524812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718524856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718524857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718524859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718524860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:04,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718524916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:04,995 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/93934fba87224534959de47df1bb136b 2023-04-17 08:01:05,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718525120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,404 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d41d7c2f8fe048faa87e0c3221922a8c 2023-04-17 08:01:05,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718525424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,812 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/deff4cff9ed548ea8a543ff7ec0547d7 2023-04-17 08:01:05,816 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/93934fba87224534959de47df1bb136b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b 2023-04-17 08:01:05,820 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b, entries=150, sequenceid=267, filesize=11.9 K 2023-04-17 08:01:05,820 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d41d7c2f8fe048faa87e0c3221922a8c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c 2023-04-17 08:01:05,823 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c, entries=150, sequenceid=267, filesize=11.9 K 2023-04-17 08:01:05,824 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/deff4cff9ed548ea8a543ff7ec0547d7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7 2023-04-17 08:01:05,827 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7, entries=150, sequenceid=267, filesize=11.9 K 2023-04-17 08:01:05,827 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a8d6cbd896bae9327132746dceb281c8 in 1240ms, sequenceid=267, compaction requested=true 2023-04-17 08:01:05,828 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:05,828 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:05,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:05,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:05,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:05,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:05,860 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:05,861 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,861 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:05,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:05,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702e201c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:05,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:05,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:05,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:01:05,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:05,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:05,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:05,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:05,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:05,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:05,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:05,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:05,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:05,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:05,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e51c4ea0bde349f099384536357e5fce 2023-04-17 08:01:05,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:05,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:05,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:05,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/0202a21ce90745edb61421ea7cc8abe6 2023-04-17 08:01:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:05,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:05,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:05,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:05,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:05,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:05,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:05,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:05,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:05,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718525884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718465877, max timestamp=1681718465889Num uniques 859; 2023-04-17 08:01:05,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718465877, max timestamp=1681718465889Num uniques 859; 2023-04-17 08:01:05,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:01:05,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:01:05,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718465877, max timestamp=1681718465889Num uniques 859; 2023-04-17 08:01:05,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:01:05,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718525909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718525910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718525913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:05,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,920 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:05,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:05,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:05,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:05,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:05,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:05,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:05,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:05,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:05,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:05,921 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:05,921 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:05,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718525927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:05,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:05,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718525991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42660 deadline: 1681718526014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718526014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718526018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,021 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:06,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:06,026 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:06,026 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,026 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:06,026 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:06,027 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,027 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:06,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:06,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:06,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:06,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:06,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:06,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:06,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d121698[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:06,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:06,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:06,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:06,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:06,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:06,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:06,064 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,064 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:06,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:06,065 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:06,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:06,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:06,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:06,065 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:06,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:06,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:06,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:06,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:06,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:06,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:06,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:06,081 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:06,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,081 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:06,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,081 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:06,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:06,181 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:06,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:06,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:06,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:06,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:06,188 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:06,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:06,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:06,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:06,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:06,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:06,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:06,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718526198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:06,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:06,208 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:06,208 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,208 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:06,215 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:06,215 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:06,216 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,216 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:06,216 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:06,216 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,216 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:06,216 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:06,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:06,217 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:06,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718526218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718526218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,227 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718526222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,227 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,228 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,228 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:06,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b256de[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:06,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:06,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:06,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:06,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:06,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:06,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:06,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:06,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:06,249 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:06,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:06,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:06,250 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:06,250 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:06,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:06,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:06,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:06,256 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:06,269 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:06,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:06,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:06,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,269 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:06,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,269 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:06,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:06,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/3ef3b55d333a47b08e9436dc6a8583ae 2023-04-17 08:01:06,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e51c4ea0bde349f099384536357e5fce as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce 2023-04-17 08:01:06,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce, entries=200, sequenceid=279, filesize=14.3 K 2023-04-17 08:01:06,304 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/0202a21ce90745edb61421ea7cc8abe6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6 2023-04-17 08:01:06,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6, entries=150, sequenceid=279, filesize=11.9 K 2023-04-17 08:01:06,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/3ef3b55d333a47b08e9436dc6a8583ae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae 2023-04-17 08:01:06,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae, entries=150, sequenceid=279, filesize=11.9 K 2023-04-17 08:01:06,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for a8d6cbd896bae9327132746dceb281c8 in 451ms, sequenceid=279, compaction requested=true 2023-04-17 08:01:06,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:06,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:06,317 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:01:06,317 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:01:06,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:06,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:06,319 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75298 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:01:06,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72858 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:01:06,319 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:06,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:06,319 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,319 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,319 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/647ec4c36738436ab78bb200c1fa949b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=71.2 K 2023-04-17 08:01:06,319 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/dc2fbaebf84f44679f2f8c6fb4965f9d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=73.5 K 2023-04-17 08:01:06,319 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 647ec4c36738436ab78bb200c1fa949b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459418 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5e911f833d314cd884dcd4efb2421f7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681718460394 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting dc2fbaebf84f44679f2f8c6fb4965f9d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459418 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8319c9fd4a75453b93437ef923427c7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681718461565 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 85d54683fdca4f87a6fc793b39d16ec1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681718460394 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 394e5880c88a44998569a6139b02f4a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681718462656 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9348febbcbbd4be2ae627b8d9db51459, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681718461565 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d41d7c2f8fe048faa87e0c3221922a8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681718463739 2023-04-17 08:01:06,320 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e90996d32b7a4c67a37c110ac8c97ed8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681718462656 2023-04-17 08:01:06,321 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0202a21ce90745edb61421ea7cc8abe6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:06,321 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 93934fba87224534959de47df1bb136b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681718463739 2023-04-17 08:01:06,321 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e51c4ea0bde349f099384536357e5fce, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:06,329 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:06,332 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#246 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:06,370 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:06,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:06,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:06,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:06,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:06,372 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:06,372 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,372 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:06,372 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:06,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:06,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:06,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:06,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:06,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:06,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:06,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:06,382 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:06,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:01:06,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:06,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718526525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718526526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718526527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718526530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718526630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718526630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718526630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,744 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/e09e64dabf634bdbaf573e7963c074ae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e09e64dabf634bdbaf573e7963c074ae 2023-04-17 08:01:06,746 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/e8058896064e4fd79c1d6737f85aa302 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e8058896064e4fd79c1d6737f85aa302 2023-04-17 08:01:06,750 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into e09e64dabf634bdbaf573e7963c074ae(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:06,750 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:06,750 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=10, startTime=1681718466317; duration=0sec 2023-04-17 08:01:06,750 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into e8058896064e4fd79c1d6737f85aa302(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:06,750 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:06,750 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:06,750 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:01:06,750 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=10, startTime=1681718466317; duration=0sec 2023-04-17 08:01:06,750 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:06,751 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72858 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:01:06,752 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:06,752 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:06,752 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/38f0276f61b44858bb61257d5141f4bd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=71.2 K 2023-04-17 08:01:06,752 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 38f0276f61b44858bb61257d5141f4bd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681718459418 2023-04-17 08:01:06,752 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f5343d9e9c3e4081996f79212fe57225, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681718460394 2023-04-17 08:01:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2722558cbe7243818b213ade6b5975ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681718461565 2023-04-17 08:01:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 972716e12a2c474aa31249a4bd68399a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681718462656 2023-04-17 08:01:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting deff4cff9ed548ea8a543ff7ec0547d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681718463739 2023-04-17 08:01:06,754 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3ef3b55d333a47b08e9436dc6a8583ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:06,762 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:06,772 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/a4d510ecab904390a184e7723a346454 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a4d510ecab904390a184e7723a346454 2023-04-17 08:01:06,777 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into a4d510ecab904390a184e7723a346454(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:06,777 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:06,777 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=10, startTime=1681718466317; duration=0sec 2023-04-17 08:01:06,777 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:06,788 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/0c8dda8de6884bb98ecea2cc294313e7 2023-04-17 08:01:06,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42662 deadline: 1681718526834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718526834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718526834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:06,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718526930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42714 deadline: 1681718527034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718527136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718527137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718527138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,198 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/3c2c95a874f4417986ee13ca8de2826d 2023-04-17 08:01:07,611 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/ca01325ab61b43fdbd6e7e4498f710a4 2023-04-17 08:01:07,616 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/0c8dda8de6884bb98ecea2cc294313e7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7 2023-04-17 08:01:07,619 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7, entries=150, sequenceid=313, filesize=11.9 K 2023-04-17 08:01:07,619 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/3c2c95a874f4417986ee13ca8de2826d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d 2023-04-17 08:01:07,623 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d, entries=150, sequenceid=313, filesize=11.9 K 2023-04-17 08:01:07,624 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/ca01325ab61b43fdbd6e7e4498f710a4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4 2023-04-17 08:01:07,629 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4, entries=150, sequenceid=313, filesize=11.9 K 2023-04-17 08:01:07,629 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for a8d6cbd896bae9327132746dceb281c8 in 1247ms, sequenceid=313, compaction requested=false 2023-04-17 08:01:07,629 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:07,629 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:07,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:07,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:07,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:07,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:07,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:07,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:07,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/446276c270d84a23b1024965f250a47b 2023-04-17 08:01:07,660 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,660 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:07,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5575d16a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:07,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:07,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718527662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718527664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718527665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,672 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:07,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:07,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:07,672 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,672 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:07,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:07,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:07,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:07,682 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:07,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:07,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:07,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:07,682 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:07,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,689 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,689 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:07,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:07,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:07,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,713 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:07,713 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:07,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718527766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718527767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718527768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,813 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:07,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:07,816 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:07,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:07,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:07,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:07,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,817 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,827 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:07,827 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:07,827 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,827 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:07,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:07,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:07,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:07,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:07,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:07,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:07,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:07,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:07,828 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:07,829 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:07,829 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:07,829 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:07,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:07,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:07,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:07,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:07,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,838 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,839 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:07,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54da2518[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:07,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:07,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,847 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:07,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:07,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:07,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:07,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:07,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:07,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:07,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:07,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:07,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:07,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:07,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:07,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:07,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:07,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:07,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:07,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:07,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,889 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:07,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:07,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:07,889 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:07,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718527967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718527970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42658 deadline: 1681718527970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:07,990 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:07,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:07,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:07,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:07,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:07,992 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:07,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:07,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:07,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:07,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:07,993 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,002 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,002 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:08,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:08,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:08,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:08,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:08,004 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:08,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:08,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,013 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:08,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d15fcd[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:08,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:08,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:08,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:08,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:08,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:08,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:08,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:08,033 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:08,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:08,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:08,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:08,033 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:08,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718528039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:08,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,047 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:08,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,047 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/8849020c023f49e7a13fd21f7ba6c5b2 2023-04-17 08:01:08,147 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:08,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:08,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:08,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:08,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:08,150 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:08,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:08,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:08,151 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,151 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,160 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:08,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:08,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:08,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:08,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:08,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:08,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:08,172 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c4e28f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:08,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:08,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:08,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:08,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:08,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:08,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:08,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,188 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:08,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:08,191 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:08,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:08,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:08,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:08,191 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:08,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:08,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:08,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,206 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:08,206 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:08,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718528271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718528274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718528274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,306 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:08,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:08,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:08,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:08,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:08,309 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:08,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:08,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:08,310 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,310 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,335 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,335 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,335 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,335 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:08,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:08,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:08,336 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:08,336 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,336 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:08,337 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,337 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:08,337 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:08,337 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:08,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:08,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:08,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1341807c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:08,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:08,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,355 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:08,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:08,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:08,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:08,363 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,363 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,364 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,364 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:08,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:08,365 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:08,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:08,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:08,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:08,366 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:08,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,372 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:08,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,372 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:08,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:08,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:08,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:08,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:08,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,380 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:08,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:08,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,380 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:08,381 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,381 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:08,381 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:08,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/3f83780cf001428999e11a75424f454b 2023-04-17 08:01:08,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/446276c270d84a23b1024965f250a47b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b 2023-04-17 08:01:08,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b, entries=200, sequenceid=330, filesize=14.3 K 2023-04-17 08:01:08,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/8849020c023f49e7a13fd21f7ba6c5b2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2 2023-04-17 08:01:08,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2, entries=150, sequenceid=330, filesize=11.9 K 2023-04-17 08:01:08,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/3f83780cf001428999e11a75424f454b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b 2023-04-17 08:01:08,481 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:08,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:08,484 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:08,484 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,484 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:08,484 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:08,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b, entries=150, sequenceid=330, filesize=11.9 K 2023-04-17 08:01:08,485 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,485 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a8d6cbd896bae9327132746dceb281c8 in 841ms, sequenceid=330, compaction requested=true 2023-04-17 08:01:08,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:08,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:08,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:08,486 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:08,486 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:08,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:08,486 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:08,487 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:08,487 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e8058896064e4fd79c1d6737f85aa302, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=38.6 K 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e8058896064e4fd79c1d6737f85aa302, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:08,487 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,487 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e09e64dabf634bdbaf573e7963c074ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.2 K 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 0c8dda8de6884bb98ecea2cc294313e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681718465906 2023-04-17 08:01:08,487 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e09e64dabf634bdbaf573e7963c074ae, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:08,488 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3c2c95a874f4417986ee13ca8de2826d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681718465906 2023-04-17 08:01:08,488 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8849020c023f49e7a13fd21f7ba6c5b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:08,491 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 446276c270d84a23b1024965f250a47b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:08,492 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:08,494 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,494 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:08,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:08,494 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,494 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:08,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:08,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:08,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:08,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:08,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:08,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:08,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:08,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:08,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:08,496 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:08,497 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:08,500 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:08,512 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/68f0545cb694496695a359c4e24a613f 2023-04-17 08:01:08,520 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/c5b5084760864b05ac8135c0e0f9968b 2023-04-17 08:01:08,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:08,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718528817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718528817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718528817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,905 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/5448e12e1a70423cadd31bdebbca34ae as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5448e12e1a70423cadd31bdebbca34ae 2023-04-17 08:01:08,909 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into 5448e12e1a70423cadd31bdebbca34ae(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:08,909 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:08,909 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=13, startTime=1681718468486; duration=0sec 2023-04-17 08:01:08,909 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:08,909 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:08,910 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:08,910 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:08,910 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,911 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a4d510ecab904390a184e7723a346454, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.2 K 2023-04-17 08:01:08,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a4d510ecab904390a184e7723a346454, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681718464805 2023-04-17 08:01:08,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ca01325ab61b43fdbd6e7e4498f710a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681718465906 2023-04-17 08:01:08,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3f83780cf001428999e11a75424f454b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:08,918 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/ea09eb06ddcc41ae9f6ec12c8de50aef as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ea09eb06ddcc41ae9f6ec12c8de50aef 2023-04-17 08:01:08,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718528920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718528920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,924 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into ea09eb06ddcc41ae9f6ec12c8de50aef(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:08,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,924 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:08,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718528924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,924 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=13, startTime=1681718468486; duration=0sec 2023-04-17 08:01:08,924 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:08,929 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:08,938 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f803b65e222145c3b143d9bfc5918202 2023-04-17 08:01:08,942 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/68f0545cb694496695a359c4e24a613f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f 2023-04-17 08:01:08,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:08,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718528944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,947 DEBUG [Thread-1612] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:08,947 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f, entries=150, sequenceid=348, filesize=11.9 K 2023-04-17 08:01:08,948 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/c5b5084760864b05ac8135c0e0f9968b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b 2023-04-17 08:01:08,951 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b, entries=150, sequenceid=348, filesize=11.9 K 2023-04-17 08:01:08,952 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/f803b65e222145c3b143d9bfc5918202 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202 2023-04-17 08:01:08,957 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202, entries=150, sequenceid=348, filesize=11.9 K 2023-04-17 08:01:08,958 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for a8d6cbd896bae9327132746dceb281c8 in 462ms, sequenceid=348, compaction requested=false 2023-04-17 08:01:08,958 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:08,958 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:08,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:08,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:08,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:08,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:08,958 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,010 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,010 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,011 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,011 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@163685e9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,047 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,047 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,047 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:09,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:09,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:09,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718529138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718529138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718529140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,147 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:09,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:09,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:09,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:09,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:09,150 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:09,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:09,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:09,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,150 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,160 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,161 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:09,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:09,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:09,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:09,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:09,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:09,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,172 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656a9fd5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,184 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,185 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,185 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,185 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,194 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,194 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,194 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,200 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,201 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,210 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,210 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,211 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,211 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,222 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,223 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,223 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:09,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718529243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718529243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718529243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,323 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:09,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:09,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:09,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:09,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:09,328 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:09,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:09,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:09,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,339 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,340 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/b341e7cd1bd6467bb439c9d2ee2c1b70 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/b341e7cd1bd6467bb439c9d2ee2c1b70 2023-04-17 08:01:09,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:09,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:09,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:09,341 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:09,341 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,342 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:09,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:09,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:09,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:09,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:09,344 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,344 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:09,345 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into b341e7cd1bd6467bb439c9d2ee2c1b70(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:09,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:09,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,345 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:09,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:09,345 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=13, startTime=1681718468486; duration=0sec 2023-04-17 08:01:09,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,345 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:09,355 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a75d96d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,386 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,386 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,386 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,386 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,399 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,399 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718529446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718529446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718529446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,469 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,469 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,469 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,469 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/ccb2d76161ff479b97e3fedada75b98a 2023-04-17 08:01:09,569 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:09,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:09,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:09,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:09,572 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:09,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:09,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:09,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:09,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:09,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:09,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:09,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:09,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:09,581 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:09,581 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,582 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:09,582 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:09,582 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:09,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:09,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:09,582 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a8ee3f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,607 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,607 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,622 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,622 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,722 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:09,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:09,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:09,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:09,725 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:09,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:09,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:09,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,726 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,735 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,736 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,736 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:09,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:09,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:09,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:09,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:09,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,747 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,747 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,747 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17133cae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718529750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718529750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:09,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718529751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,755 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,756 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,765 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,772 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,772 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,772 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:09,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,781 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,781 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,781 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,881 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:09,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:09,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:09,891 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:09,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:09,891 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:09,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,892 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:09,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:09,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:09,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:09,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:09,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:09,899 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:09,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:09,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:09,903 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,903 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:09,903 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:09,903 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:09,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:09,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b68dae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:09,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:09,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:09,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:09,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:09,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:09,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/e5ce48ead0bd4073b3c9e0c3afa83ad2 2023-04-17 08:01:09,949 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:09,949 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,949 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:09,949 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/2d356c4a188548d7a8fc70d52db56087 2023-04-17 08:01:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:09,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:09,951 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:09,951 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:09,952 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:09,952 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:09,952 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:09,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/ccb2d76161ff479b97e3fedada75b98a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a 2023-04-17 08:01:09,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a, entries=250, sequenceid=368, filesize=16.7 K 2023-04-17 08:01:09,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/e5ce48ead0bd4073b3c9e0c3afa83ad2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2 2023-04-17 08:01:09,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 08:01:09,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/2d356c4a188548d7a8fc70d52db56087 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087 2023-04-17 08:01:09,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:09,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,964 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a8d6cbd896bae9327132746dceb281c8 in 840ms, sequenceid=368, compaction requested=true 2023-04-17 08:01:09,964 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:09,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:09,964 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,964 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:09,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:09,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,965 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42086 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:09,965 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:09,965 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:09,966 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,966 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:09,966 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,966 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ea09eb06ddcc41ae9f6ec12c8de50aef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=41.1 K 2023-04-17 08:01:09,966 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5448e12e1a70423cadd31bdebbca34ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.3 K 2023-04-17 08:01:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,966 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ea09eb06ddcc41ae9f6ec12c8de50aef, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:09,966 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 5448e12e1a70423cadd31bdebbca34ae, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:09,966 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 68f0545cb694496695a359c4e24a613f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681718467652 2023-04-17 08:01:09,967 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c5b5084760864b05ac8135c0e0f9968b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681718467652 2023-04-17 08:01:09,967 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ccb2d76161ff479b97e3fedada75b98a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,967 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e5ce48ead0bd4073b3c9e0c3afa83ad2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:09,972 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:09,973 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#264 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:09,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:09,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:09,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:09,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:09,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:09,981 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:09,981 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:09,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:09,989 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/89392eb9da464cb98c0f06d035b09320 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/89392eb9da464cb98c0f06d035b09320 2023-04-17 08:01:09,990 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/c2b727108e7a4d7db0723008f9d8e12f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c2b727108e7a4d7db0723008f9d8e12f 2023-04-17 08:01:09,994 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into 89392eb9da464cb98c0f06d035b09320(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:09,994 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:09,994 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=13, startTime=1681718469964; duration=0sec 2023-04-17 08:01:09,994 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:09,994 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:09,994 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into c2b727108e7a4d7db0723008f9d8e12f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:09,994 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:09,994 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=13, startTime=1681718469964; duration=0sec 2023-04-17 08:01:09,994 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:09,995 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:09,995 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:09,995 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:09,995 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/b341e7cd1bd6467bb439c9d2ee2c1b70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.3 K 2023-04-17 08:01:09,996 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b341e7cd1bd6467bb439c9d2ee2c1b70, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681718466525 2023-04-17 08:01:09,996 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f803b65e222145c3b143d9bfc5918202, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681718467652 2023-04-17 08:01:09,996 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2d356c4a188548d7a8fc70d52db56087, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:10,001 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#265 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:10,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:10,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:10,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,081 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:10,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:10,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:10,089 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:10,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:10,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:10,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718530096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:10,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:10,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:10,099 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,099 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:10,099 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:10,099 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:10,099 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:10,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:10,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220c96a4[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:10,108 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:10,108 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:10,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:10,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:10,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:10,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:10,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:10,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:10,124 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:10,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:10,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:10,125 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:10,125 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:10,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,131 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:10,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,139 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:10,139 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:10,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,139 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718530199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,239 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:10,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:10,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:10,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:10,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:10,242 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:10,242 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,242 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:10,242 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:10,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,243 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,252 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,252 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,252 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:10,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:10,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:10,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:10,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:10,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:10,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:10,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:10,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:10,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718530254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718530254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718530255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:10,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a74dcc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:10,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:10,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:10,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:10,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:10,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:10,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:10,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:10,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:10,282 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:10,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:10,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:10,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:10,282 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:10,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:10,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:10,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,297 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,297 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:10,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,398 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:10,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:10,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:10,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:10,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:10,400 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:10,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:10,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:10,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718530402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,408 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/48c741291a554ef8b7689405610d1ba7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/48c741291a554ef8b7689405610d1ba7 2023-04-17 08:01:10,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,410 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,411 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,411 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,412 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 48c741291a554ef8b7689405610d1ba7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:10,412 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:10,412 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=13, startTime=1681718469965; duration=0sec 2023-04-17 08:01:10,412 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:10,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:10,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:10,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:10,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:10,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:10,413 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:10,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:10,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:10,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e5fed3a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:10,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:10,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:10,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:10,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:10,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:10,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:10,439 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,439 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:10,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:10,440 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:10,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:10,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:10,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:10,440 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:10,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/332ee33d99024a27abcd2634e08b07bc 2023-04-17 08:01:10,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b 2023-04-17 08:01:10,476 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,476 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:10,476 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:10,477 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,478 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/66d671b3efc54946b669b361693c5bb4 2023-04-17 08:01:10,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/332ee33d99024a27abcd2634e08b07bc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc 2023-04-17 08:01:10,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc, entries=200, sequenceid=386, filesize=14.3 K 2023-04-17 08:01:10,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b 2023-04-17 08:01:10,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b, entries=150, sequenceid=386, filesize=11.9 K 2023-04-17 08:01:10,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/66d671b3efc54946b669b361693c5bb4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4 2023-04-17 08:01:10,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4, entries=150, sequenceid=386, filesize=11.9 K 2023-04-17 08:01:10,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a8d6cbd896bae9327132746dceb281c8 in 452ms, sequenceid=386, compaction requested=false 2023-04-17 08:01:10,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:10,502 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,502 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:10,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:10,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,502 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:10,503 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:10,502 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:10,502 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:10,502 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:10,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:10,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:10,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,603 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:10,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:10,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:10,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:10,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:10,605 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:10,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:10,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:10,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,614 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:10,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:10,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:10,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:10,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:10,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:10,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:10,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:10,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:10,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:10,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:10,615 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:10,615 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:10,624 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/fcc4ce7383c94a5d8d19347a3e75f35a 2023-04-17 08:01:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:10,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718530736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:10,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:10,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718530840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,033 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d36c80c678f7479b83b5b60567ff75cb 2023-04-17 08:01:11,042 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/628b29fa22b748f78c8a980efccc0a4b 2023-04-17 08:01:11,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718531042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/fcc4ce7383c94a5d8d19347a3e75f35a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a 2023-04-17 08:01:11,050 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a, entries=150, sequenceid=404, filesize=11.9 K 2023-04-17 08:01:11,051 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d36c80c678f7479b83b5b60567ff75cb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb 2023-04-17 08:01:11,054 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb, entries=150, sequenceid=404, filesize=11.9 K 2023-04-17 08:01:11,055 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/628b29fa22b748f78c8a980efccc0a4b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b 2023-04-17 08:01:11,058 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b, entries=150, sequenceid=404, filesize=11.9 K 2023-04-17 08:01:11,059 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a8d6cbd896bae9327132746dceb281c8 in 444ms, sequenceid=404, compaction requested=true 2023-04-17 08:01:11,059 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:11,059 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:11,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:11,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:11,059 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:11,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,081 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:11,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c96d0a2[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:11,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:11,082 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:11,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:11,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:11,122 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:11,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:11,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:11,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:11,134 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:11,134 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:11,134 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:11,134 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:11,134 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:11,144 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,144 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,144 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,144 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,146 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:11,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:11,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,189 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:11,189 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,189 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,190 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:11,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:11,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:11,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/7e43cba5e11f4c34bb3f512fc1b988b4 2023-04-17 08:01:11,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718531277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718531278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718531280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,289 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:11,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:11,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:11,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:11,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:11,292 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:11,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:11,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:11,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,303 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,304 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:11,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:11,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:11,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:11,306 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,306 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:11,306 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:11,306 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:11,306 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:11,313 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:11,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:11,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae9c622[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:11,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718531346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,355 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:11,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:11,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:11,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:11,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718531381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718531383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718531383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:11,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:11,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:11,401 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:11,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:11,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:11,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:11,401 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:11,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:11,457 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,457 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:11,456 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,557 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:11,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:11,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:11,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718531582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:11,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:11,586 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:11,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:11,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:11,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718531587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718531592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:11,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:11,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:11,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:11,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:11,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,599 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:11,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:11,600 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:11,600 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,600 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:11,600 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:11,600 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,601 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:11,601 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:11,601 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,601 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:11,601 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:11,610 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,610 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,611 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,611 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:11,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ca239e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:11,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:11,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:11,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:11,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:11,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:11,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:11,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:11,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:11,633 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:11,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:11,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:11,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:11,633 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:11,644 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,644 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,644 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:11,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,674 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:11,674 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,674 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,675 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,675 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/4a6ae6b9e7ae4827a1c25ff734919b74 2023-04-17 08:01:11,774 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:11,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:11,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:11,777 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:11,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:11,777 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:11,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,788 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:11,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:11,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:11,790 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,790 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:11,790 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:11,790 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:11,790 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:11,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:11,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41dd3480[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:11,831 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:11,831 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718531852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:11,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:11,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:11,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:11,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718531884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718531891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:11,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718531897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:11,899 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:11,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:11,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:11,899 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:11,899 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:11,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:11,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,919 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:11,919 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:11,920 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:11,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:11,956 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,956 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:11,956 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:11,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:11,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:11,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:11,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:11,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:11,956 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:11,956 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:12,056 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:12,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:12,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:12,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:12,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:12,059 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:12,059 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:12,059 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:12,059 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:12,060 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,060 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:12,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:12,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:12,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:12,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:12,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:12,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:12,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:12,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:12,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:12,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:12,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:12,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,091 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:12,091 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:12,091 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,091 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:12,091 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:12,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/e8723ab7e70c4ee0a36fda0760099c30 2023-04-17 08:01:12,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/7e43cba5e11f4c34bb3f512fc1b988b4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4 2023-04-17 08:01:12,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4, entries=200, sequenceid=420, filesize=14.3 K 2023-04-17 08:01:12,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/4a6ae6b9e7ae4827a1c25ff734919b74 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74 2023-04-17 08:01:12,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74, entries=150, sequenceid=420, filesize=11.9 K 2023-04-17 08:01:12,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/e8723ab7e70c4ee0a36fda0760099c30 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30 2023-04-17 08:01:12,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30, entries=150, sequenceid=420, filesize=11.9 K 2023-04-17 08:01:12,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a8d6cbd896bae9327132746dceb281c8 in 854ms, sequenceid=420, compaction requested=true 2023-04-17 08:01:12,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:12,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:12,115 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:12,115 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:12,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:12,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:12,116 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:12,116 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:12,116 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:12,116 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:12,116 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,116 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,116 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c2b727108e7a4d7db0723008f9d8e12f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=48.3 K 2023-04-17 08:01:12,116 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/89392eb9da464cb98c0f06d035b09320, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=53.1 K 2023-04-17 08:01:12,117 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c2b727108e7a4d7db0723008f9d8e12f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:12,117 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 89392eb9da464cb98c0f06d035b09320, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:12,117 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f67c5b384c1e4d9d8b3716c5cb5c3e1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681718469132 2023-04-17 08:01:12,117 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 332ee33d99024a27abcd2634e08b07bc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681718469132 2023-04-17 08:01:12,117 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d36c80c678f7479b83b5b60567ff75cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681718470080 2023-04-17 08:01:12,118 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting fcc4ce7383c94a5d8d19347a3e75f35a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681718470080 2023-04-17 08:01:12,118 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4a6ae6b9e7ae4827a1c25ff734919b74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470729 2023-04-17 08:01:12,118 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7e43cba5e11f4c34bb3f512fc1b988b4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470727 2023-04-17 08:01:12,125 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:12,127 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,128 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:12,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb4ffd9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:12,130 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:12,130 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,137 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:12,161 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/953c0ecc01a3477e9708d5587d61785a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/953c0ecc01a3477e9708d5587d61785a 2023-04-17 08:01:12,166 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into 953c0ecc01a3477e9708d5587d61785a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:12,166 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:12,166 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=12, startTime=1681718472115; duration=0sec 2023-04-17 08:01:12,166 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:12,166 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:12,168 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:12,168 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:12,168 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,168 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/48c741291a554ef8b7689405610d1ba7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=48.3 K 2023-04-17 08:01:12,168 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 48c741291a554ef8b7689405610d1ba7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718468785 2023-04-17 08:01:12,168 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 66d671b3efc54946b669b361693c5bb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681718469132 2023-04-17 08:01:12,169 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 628b29fa22b748f78c8a980efccc0a4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681718470080 2023-04-17 08:01:12,169 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e8723ab7e70c4ee0a36fda0760099c30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470729 2023-04-17 08:01:12,175 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#277 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:12,177 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,177 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:12,177 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:12,177 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:12,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:12,184 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/4cecc813bdd04ca4afa266899ad6a1cf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4cecc813bdd04ca4afa266899ad6a1cf 2023-04-17 08:01:12,187 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 4cecc813bdd04ca4afa266899ad6a1cf(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:12,188 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:12,188 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=12, startTime=1681718472115; duration=0sec 2023-04-17 08:01:12,188 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:12,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:12,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:12,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:12,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:12,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:12,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:12,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:12,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:12,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:12,235 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,235 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,236 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:12,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:12,236 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:12,252 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:12,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:12,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:12,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:12,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:12,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,252 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,253 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,253 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,253 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,253 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:12,253 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:12,253 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:12,353 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:12,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:12,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:12,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:12,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:12,355 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:12,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:12,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:12,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:12,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:12,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:12,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:12,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:12,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:12,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:12,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:12,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:12,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:12,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:12,370 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,370 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:12,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,377 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/387804a311d34d32ab58100e8216ea32 2023-04-17 08:01:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:12,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:12,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718532407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718532407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718532409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718532510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718532510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718532512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,564 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/a562b59c35d44c4ba4364b246ce1bcde as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a562b59c35d44c4ba4364b246ce1bcde 2023-04-17 08:01:12,568 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into a562b59c35d44c4ba4364b246ce1bcde(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:12,568 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:12,568 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=12, startTime=1681718472115; duration=0sec 2023-04-17 08:01:12,568 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:12,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718532711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718532712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718532714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,785 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/e19e2bcc2324413c9acd1b22fcc369d1 2023-04-17 08:01:12,793 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/156efd062d7f432ea4701b70a8029519 2023-04-17 08:01:12,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/387804a311d34d32ab58100e8216ea32 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32 2023-04-17 08:01:12,800 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32, entries=150, sequenceid=438, filesize=11.9 K 2023-04-17 08:01:12,801 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/e19e2bcc2324413c9acd1b22fcc369d1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1 2023-04-17 08:01:12,804 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1, entries=150, sequenceid=438, filesize=11.9 K 2023-04-17 08:01:12,804 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/156efd062d7f432ea4701b70a8029519 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519 2023-04-17 08:01:12,807 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519, entries=150, sequenceid=438, filesize=11.9 K 2023-04-17 08:01:12,808 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a8d6cbd896bae9327132746dceb281c8 in 438ms, sequenceid=438, compaction requested=false 2023-04-17 08:01:12,808 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:12,808 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:12,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:12,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:12,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:12,808 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:12,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,847 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:12,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f28e65b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:12,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:12,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:12,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:12,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:12,861 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:12,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:12,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:12,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:12,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:12,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:12,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:12,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:12,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:12,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:12,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:12,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:12,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:12,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:12,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:12,898 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:12,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718532899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:12,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:12,914 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:12,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:12,914 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:12,914 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:12,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:42736 deadline: 1681718532976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:12,978 DEBUG [Thread-1612] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:13,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718533002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,014 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:13,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:13,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718533014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718533016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:13,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718533017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:13,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:13,020 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:13,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:13,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:13,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:13,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:13,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:13,048 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,048 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:13,048 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:13,048 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:13,048 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:13,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,065 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:13,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e4ffcf3[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:13,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:13,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:13,080 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:13,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:13,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:13,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:13,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:13,099 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:13,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:13,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:13,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:13,099 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:13,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,106 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,108 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,109 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,119 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:13,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,119 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,121 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:13,121 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:13,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718533204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,221 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:13,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:13,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:13,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:13,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:13,224 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:13,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:13,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:13,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:13,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:13,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:13,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:13,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:13,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:13,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:13,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:13,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:13,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:13,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:13,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:13,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:13,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@661eb88e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:13,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:13,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:13,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:13,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:13,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:13,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:13,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:13,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:13,257 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:13,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:13,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:13,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:13,257 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:13,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/838c4acb55724f8697dce6bddbc3432b 2023-04-17 08:01:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:13,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:13,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:13,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,281 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:13,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,281 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:13,281 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,381 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:13,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:13,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:13,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:13,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:13,384 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:13,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:13,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:13,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,384 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,394 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,394 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:13,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:13,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:13,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:13,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:13,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:13,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:13,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a48c9a3[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:13,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:13,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:13,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:13,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:13,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:13,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:13,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:13,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:13,423 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:13,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:13,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:13,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:13,424 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:13,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:13,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:13,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,439 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,439 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42714 deadline: 1681718533505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42662 deadline: 1681718533519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42660 deadline: 1681718533520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:13,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:42658 deadline: 1681718533520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,539 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:13,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:13,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:13,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:13,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:13,542 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:13,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:13,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:13,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,542 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:13,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:13,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:13,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:13,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:13,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:13,560 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,560 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:13,560 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:13,560 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:13,560 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:13,585 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,585 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,585 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,585 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:13,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:13,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42eae28f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:13,586 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:13,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:13,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:13,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:13,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:13,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:13,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:13,607 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:13,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:13,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:13,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:13,607 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:13,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:13,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:13,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:13,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:13,622 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:13,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,622 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:13,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d8bec2151c8c4b078ccde0417fc40e43 2023-04-17 08:01:13,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/8325e22a921d456394c3551728eb41a4 2023-04-17 08:01:13,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/838c4acb55724f8697dce6bddbc3432b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b 2023-04-17 08:01:13,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b, entries=150, sequenceid=456, filesize=11.9 K 2023-04-17 08:01:13,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/d8bec2151c8c4b078ccde0417fc40e43 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43 2023-04-17 08:01:13,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43, entries=150, sequenceid=456, filesize=11.9 K 2023-04-17 08:01:13,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/8325e22a921d456394c3551728eb41a4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4 2023-04-17 08:01:13,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4, entries=150, sequenceid=456, filesize=11.9 K 2023-04-17 08:01:13,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a8d6cbd896bae9327132746dceb281c8 in 840ms, sequenceid=456, compaction requested=true 2023-04-17 08:01:13,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:13,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:13,699 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:13,699 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:13,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:13,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:13,699 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/A is initiating minor compaction (all files) 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/B is initiating minor compaction (all files) 2023-04-17 08:01:13,700 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/A in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,700 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/B in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,700 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/953c0ecc01a3477e9708d5587d61785a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.6 K 2023-04-17 08:01:13,700 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a562b59c35d44c4ba4364b246ce1bcde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.6 K 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 953c0ecc01a3477e9708d5587d61785a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470729 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a562b59c35d44c4ba4364b246ce1bcde, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470729 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 387804a311d34d32ab58100e8216ea32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681718471276 2023-04-17 08:01:13,700 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e19e2bcc2324413c9acd1b22fcc369d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681718471276 2023-04-17 08:01:13,701 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 838c4acb55724f8697dce6bddbc3432b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681718472404 2023-04-17 08:01:13,701 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d8bec2151c8c4b078ccde0417fc40e43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681718472404 2023-04-17 08:01:13,705 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#B#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:13,707 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#A#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:13,723 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:13,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:13,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:13,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:13,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:13,725 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:13,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:13,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:13,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,735 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:13,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:13,735 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:13,735 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:13,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:13,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:13,736 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:13,736 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:13,736 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:13,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:13,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. started... 2023-04-17 08:01:13,737 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:13,737 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:13,975 DEBUG [Thread-1619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d22a55e to 127.0.0.1:62477 2023-04-17 08:01:13,975 DEBUG [Thread-1619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:13,976 DEBUG [Thread-1615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:62477 2023-04-17 08:01:13,976 DEBUG [Thread-1615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:13,976 DEBUG [Thread-1621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2179ead9 to 127.0.0.1:62477 2023-04-17 08:01:13,976 DEBUG [Thread-1621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:13,977 DEBUG [Thread-1623] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7897b06a to 127.0.0.1:62477 2023-04-17 08:01:13,977 DEBUG [Thread-1617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:62477 2023-04-17 08:01:13,977 DEBUG [Thread-1623] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:13,977 DEBUG [Thread-1617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:14,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. as already flushing 2023-04-17 08:01:14,013 DEBUG [Thread-1606] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:62477 2023-04-17 08:01:14,013 DEBUG [Thread-1606] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:14,123 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/4683d8faf6cf4fb1b889788a6f725b38 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4683d8faf6cf4fb1b889788a6f725b38 2023-04-17 08:01:14,127 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/17ad5e8fca42444394e3274ce0ed5828 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/17ad5e8fca42444394e3274ce0ed5828 2023-04-17 08:01:14,129 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/B of a8d6cbd896bae9327132746dceb281c8 into 4683d8faf6cf4fb1b889788a6f725b38(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:14,129 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:14,129 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/B, priority=13, startTime=1681718473699; duration=0sec 2023-04-17 08:01:14,129 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:14,130 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:14,131 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:14,131 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): a8d6cbd896bae9327132746dceb281c8/C is initiating minor compaction (all files) 2023-04-17 08:01:14,131 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8d6cbd896bae9327132746dceb281c8/C in TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:14,131 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4cecc813bdd04ca4afa266899ad6a1cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp, totalSize=36.6 K 2023-04-17 08:01:14,132 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4cecc813bdd04ca4afa266899ad6a1cf, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681718470729 2023-04-17 08:01:14,132 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 156efd062d7f432ea4701b70a8029519, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681718471276 2023-04-17 08:01:14,133 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8325e22a921d456394c3551728eb41a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681718472404 2023-04-17 08:01:14,134 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/A of a8d6cbd896bae9327132746dceb281c8 into 17ad5e8fca42444394e3274ce0ed5828(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:14,134 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:14,134 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/A, priority=13, startTime=1681718473699; duration=0sec 2023-04-17 08:01:14,134 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:14,139 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): a8d6cbd896bae9327132746dceb281c8#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:14,154 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/4cb45fa19d1940959f922baced64db05 2023-04-17 08:01:14,529 DEBUG [Thread-1604] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:62477 2023-04-17 08:01:14,529 DEBUG [Thread-1604] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:14,530 DEBUG [Thread-1608] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a8a575d to 127.0.0.1:62477 2023-04-17 08:01:14,530 DEBUG [Thread-1608] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:14,532 DEBUG [Thread-1610] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6347f2e6 to 127.0.0.1:62477 2023-04-17 08:01:14,533 DEBUG [Thread-1610] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:14,553 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/4f4c3a4773334d4b8c6e18d9b87996a7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4f4c3a4773334d4b8c6e18d9b87996a7 2023-04-17 08:01:14,559 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8d6cbd896bae9327132746dceb281c8/C of a8d6cbd896bae9327132746dceb281c8 into 4f4c3a4773334d4b8c6e18d9b87996a7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:14,559 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:14,559 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8., storeName=a8d6cbd896bae9327132746dceb281c8/C, priority=13, startTime=1681718473699; duration=0sec 2023-04-17 08:01:14,560 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:14,564 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/c551e8e68b7e4df6991fa8429e5813b7 2023-04-17 08:01:14,975 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/1d59e126519f40e9978ce81d67c80490 2023-04-17 08:01:14,981 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/4cb45fa19d1940959f922baced64db05 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4cb45fa19d1940959f922baced64db05 2023-04-17 08:01:14,985 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4cb45fa19d1940959f922baced64db05, entries=150, sequenceid=471, filesize=11.9 K 2023-04-17 08:01:14,986 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/c551e8e68b7e4df6991fa8429e5813b7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c551e8e68b7e4df6991fa8429e5813b7 2023-04-17 08:01:14,991 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c551e8e68b7e4df6991fa8429e5813b7, entries=150, sequenceid=471, filesize=11.9 K 2023-04-17 08:01:14,991 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/1d59e126519f40e9978ce81d67c80490 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/1d59e126519f40e9978ce81d67c80490 2023-04-17 08:01:14,995 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/1d59e126519f40e9978ce81d67c80490, entries=150, sequenceid=471, filesize=11.9 K 2023-04-17 08:01:14,996 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=26.84 KB/27480 for a8d6cbd896bae9327132746dceb281c8 in 1259ms, sequenceid=471, compaction requested=false 2023-04-17 08:01:14,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:14,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:14,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:14,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:14,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:14,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:14,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:15,044 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,044 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:15,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:15,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:15,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73733645[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:15,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:15,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,061 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:15,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:15,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:15,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:15,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:15,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:15,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:15,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:15,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:15,082 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:15,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:15,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:15,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:15,083 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:15,094 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,094 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:15,094 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:15,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:15,095 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,110 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:15,110 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:15,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:15,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:15,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:15,111 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:15,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:15,112 DEBUG [Thread-1614] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:15,112 DEBUG [Thread-1614] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:15,111 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:15,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:15,212 DEBUG [Thread-1614] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:23,054 DEBUG [Thread-1612] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36523181 to 127.0.0.1:62477 2023-04-17 08:01:23,054 DEBUG [Thread-1612] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:23,054 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:01:23,054 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 6043 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5993 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5915 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 6093 2023-04-17 08:01:23,055 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5995 2023-04-17 08:01:23,056 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:01:23,056 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:01:23,056 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:62477 2023-04-17 08:01:23,056 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:23,057 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:01:23,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:01:23,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:23,063 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718483063"}]},"ts":"1681718483063"} 2023-04-17 08:01:23,064 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:01:23,102 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:01:23,103 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, UNASSIGN}] 2023-04-17 08:01:23,106 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, UNASSIGN 2023-04-17 08:01:23,108 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a8d6cbd896bae9327132746dceb281c8, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:23,108 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718483108"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718483108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718483108"}]},"ts":"1681718483108"} 2023-04-17 08:01:23,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:01:23,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:23,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:23,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing a8d6cbd896bae9327132746dceb281c8, disabling compactions & flushes 2023-04-17 08:01:23,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:23,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:23,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. after waiting 0 ms 2023-04-17 08:01:23,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:23,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing a8d6cbd896bae9327132746dceb281c8 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-17 08:01:23,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=A 2023-04-17 08:01:23,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:23,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=B 2023-04-17 08:01:23,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:23,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8d6cbd896bae9327132746dceb281c8, store=C 2023-04-17 08:01:23,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:23,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:23,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:23,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/3b1a816a33c3481f9369a79087a5887e 2023-04-17 08:01:24,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/72256fc13ac54efc8b85274d3d5177fe 2023-04-17 08:01:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:24,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/edc65ec4ca754546ab3d67974b4b256a 2023-04-17 08:01:24,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/A/3b1a816a33c3481f9369a79087a5887e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b1a816a33c3481f9369a79087a5887e 2023-04-17 08:01:24,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b1a816a33c3481f9369a79087a5887e, entries=150, sequenceid=482, filesize=11.9 K 2023-04-17 08:01:24,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/B/72256fc13ac54efc8b85274d3d5177fe as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/72256fc13ac54efc8b85274d3d5177fe 2023-04-17 08:01:24,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/72256fc13ac54efc8b85274d3d5177fe, entries=150, sequenceid=482, filesize=11.9 K 2023-04-17 08:01:24,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/.tmp/C/edc65ec4ca754546ab3d67974b4b256a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/edc65ec4ca754546ab3d67974b4b256a 2023-04-17 08:01:24,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/edc65ec4ca754546ab3d67974b4b256a, entries=150, sequenceid=482, filesize=11.9 K 2023-04-17 08:01:24,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for a8d6cbd896bae9327132746dceb281c8 in 1274ms, sequenceid=482, compaction requested=true 2023-04-17 08:01:24,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/f9ca425ee1924ce08ceb3b636901474c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4bb5a4aca99f4c88b43295de959a4643, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/dc2fbaebf84f44679f2f8c6fb4965f9d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e8058896064e4fd79c1d6737f85aa302, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ea09eb06ddcc41ae9f6ec12c8de50aef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/89392eb9da464cb98c0f06d035b09320, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/953c0ecc01a3477e9708d5587d61785a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b] to archive 2023-04-17 08:01:24,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:24,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7f89d0aa9f924a6bbbd824d783604bf4 2023-04-17 08:01:24,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/6a4f868e94f24655bd8377f1a02e4e88 2023-04-17 08:01:24,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/f9ca425ee1924ce08ceb3b636901474c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/f9ca425ee1924ce08ceb3b636901474c 2023-04-17 08:01:24,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e12fd70e7aaf402b870969912e2e85ad 2023-04-17 08:01:24,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4119682511a2427495c4dfa261bbc637 2023-04-17 08:01:24,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/91271b071d8847ee85465dc0e0eba9d9 2023-04-17 08:01:24,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4bb5a4aca99f4c88b43295de959a4643 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4bb5a4aca99f4c88b43295de959a4643 2023-04-17 08:01:24,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b920ac473c94e51919e0ebcc9e7c89c 2023-04-17 08:01:24,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fe3fb90b533e407c9c8bc57d146e7029 2023-04-17 08:01:24,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/a3d77e1460cb435590c823eb8d3c9222 2023-04-17 08:01:24,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/dc2fbaebf84f44679f2f8c6fb4965f9d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/dc2fbaebf84f44679f2f8c6fb4965f9d 2023-04-17 08:01:24,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/85d54683fdca4f87a6fc793b39d16ec1 2023-04-17 08:01:24,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/9348febbcbbd4be2ae627b8d9db51459 2023-04-17 08:01:24,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e90996d32b7a4c67a37c110ac8c97ed8 2023-04-17 08:01:24,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/93934fba87224534959de47df1bb136b 2023-04-17 08:01:24,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e51c4ea0bde349f099384536357e5fce 2023-04-17 08:01:24,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e8058896064e4fd79c1d6737f85aa302 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/e8058896064e4fd79c1d6737f85aa302 2023-04-17 08:01:24,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/0c8dda8de6884bb98ecea2cc294313e7 2023-04-17 08:01:24,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/446276c270d84a23b1024965f250a47b 2023-04-17 08:01:24,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ea09eb06ddcc41ae9f6ec12c8de50aef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ea09eb06ddcc41ae9f6ec12c8de50aef 2023-04-17 08:01:24,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/68f0545cb694496695a359c4e24a613f 2023-04-17 08:01:24,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/ccb2d76161ff479b97e3fedada75b98a 2023-04-17 08:01:24,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/89392eb9da464cb98c0f06d035b09320 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/89392eb9da464cb98c0f06d035b09320 2023-04-17 08:01:24,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/332ee33d99024a27abcd2634e08b07bc 2023-04-17 08:01:24,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/fcc4ce7383c94a5d8d19347a3e75f35a 2023-04-17 08:01:24,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/7e43cba5e11f4c34bb3f512fc1b988b4 2023-04-17 08:01:24,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/953c0ecc01a3477e9708d5587d61785a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/953c0ecc01a3477e9708d5587d61785a 2023-04-17 08:01:24,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/387804a311d34d32ab58100e8216ea32 2023-04-17 08:01:24,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/838c4acb55724f8697dce6bddbc3432b 2023-04-17 08:01:24,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41cb64abac94c6aa7a3866a72bf2cd0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4766ba7db1f84a81ac72e279e05294f3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/647ec4c36738436ab78bb200c1fa949b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e09e64dabf634bdbaf573e7963c074ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5448e12e1a70423cadd31bdebbca34ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c2b727108e7a4d7db0723008f9d8e12f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a562b59c35d44c4ba4364b246ce1bcde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43] to archive 2023-04-17 08:01:24,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:24,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5744695b8c4847948df6879d3ed3cdf3 2023-04-17 08:01:24,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/6970a6d4fee44d2db6c80a753ccbde9f 2023-04-17 08:01:24,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41cb64abac94c6aa7a3866a72bf2cd0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41cb64abac94c6aa7a3866a72bf2cd0 2023-04-17 08:01:24,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/df57b4709be84429b69a96a66f59471e 2023-04-17 08:01:24,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a7ee43afbd294053977e38db699f8584 2023-04-17 08:01:24,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4766ba7db1f84a81ac72e279e05294f3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4766ba7db1f84a81ac72e279e05294f3 2023-04-17 08:01:24,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b655515fb658452195618c406b4d5cb0 2023-04-17 08:01:24,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/176be35b894d4354b07cc3b263d38440 2023-04-17 08:01:24,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/28ae1d8db410420ba47d17ce847c7e22 2023-04-17 08:01:24,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/647ec4c36738436ab78bb200c1fa949b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/647ec4c36738436ab78bb200c1fa949b 2023-04-17 08:01:24,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/b66ad99d33ae442c88e8de71ecb861e5 2023-04-17 08:01:24,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5e911f833d314cd884dcd4efb2421f7f 2023-04-17 08:01:24,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8319c9fd4a75453b93437ef923427c7e 2023-04-17 08:01:24,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/394e5880c88a44998569a6139b02f4a8 2023-04-17 08:01:24,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d41d7c2f8fe048faa87e0c3221922a8c 2023-04-17 08:01:24,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e09e64dabf634bdbaf573e7963c074ae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e09e64dabf634bdbaf573e7963c074ae 2023-04-17 08:01:24,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/0202a21ce90745edb61421ea7cc8abe6 2023-04-17 08:01:24,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/3c2c95a874f4417986ee13ca8de2826d 2023-04-17 08:01:24,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5448e12e1a70423cadd31bdebbca34ae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/5448e12e1a70423cadd31bdebbca34ae 2023-04-17 08:01:24,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/8849020c023f49e7a13fd21f7ba6c5b2 2023-04-17 08:01:24,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c5b5084760864b05ac8135c0e0f9968b 2023-04-17 08:01:24,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c2b727108e7a4d7db0723008f9d8e12f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c2b727108e7a4d7db0723008f9d8e12f 2023-04-17 08:01:24,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e5ce48ead0bd4073b3c9e0c3afa83ad2 2023-04-17 08:01:24,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/f67c5b384c1e4d9d8b3716c5cb5c3e1b 2023-04-17 08:01:24,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d36c80c678f7479b83b5b60567ff75cb 2023-04-17 08:01:24,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a562b59c35d44c4ba4364b246ce1bcde to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/a562b59c35d44c4ba4364b246ce1bcde 2023-04-17 08:01:24,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4a6ae6b9e7ae4827a1c25ff734919b74 2023-04-17 08:01:24,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/e19e2bcc2324413c9acd1b22fcc369d1 2023-04-17 08:01:24,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/d8bec2151c8c4b078ccde0417fc40e43 2023-04-17 08:01:24,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/26570ec414ad4eeaafbe28745632586c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8dac69cfd26f4a9a8ee8c106290b0d23, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/38f0276f61b44858bb61257d5141f4bd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a4d510ecab904390a184e7723a346454, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/b341e7cd1bd6467bb439c9d2ee2c1b70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/48c741291a554ef8b7689405610d1ba7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4cecc813bdd04ca4afa266899ad6a1cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4] to archive 2023-04-17 08:01:24,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:24,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4963d2339deb4a05b7877c7d7d95382d 2023-04-17 08:01:24,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/99d7b13793794f36a8b9a9ff23e0a0ec 2023-04-17 08:01:24,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/26570ec414ad4eeaafbe28745632586c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/26570ec414ad4eeaafbe28745632586c 2023-04-17 08:01:24,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a0f668eb3fae4deebae17fd6bb2ee20d 2023-04-17 08:01:24,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ab468f707bfd4282b6847eaddf2a3c78 2023-04-17 08:01:24,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8dac69cfd26f4a9a8ee8c106290b0d23 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8dac69cfd26f4a9a8ee8c106290b0d23 2023-04-17 08:01:24,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e874854a9569433885db48b58a9821d9 2023-04-17 08:01:24,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8398b6c66a314dd09f7f4de1c2836fd9 2023-04-17 08:01:24,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/6de3d060cdcc4ea39eeea85bae51bd16 2023-04-17 08:01:24,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/38f0276f61b44858bb61257d5141f4bd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/38f0276f61b44858bb61257d5141f4bd 2023-04-17 08:01:24,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f9ff20ffd8f64b5bb864ea5827f4b49f 2023-04-17 08:01:24,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f5343d9e9c3e4081996f79212fe57225 2023-04-17 08:01:24,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2722558cbe7243818b213ade6b5975ae 2023-04-17 08:01:24,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/972716e12a2c474aa31249a4bd68399a 2023-04-17 08:01:24,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/deff4cff9ed548ea8a543ff7ec0547d7 2023-04-17 08:01:24,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a4d510ecab904390a184e7723a346454 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/a4d510ecab904390a184e7723a346454 2023-04-17 08:01:24,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3ef3b55d333a47b08e9436dc6a8583ae 2023-04-17 08:01:24,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/ca01325ab61b43fdbd6e7e4498f710a4 2023-04-17 08:01:24,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/b341e7cd1bd6467bb439c9d2ee2c1b70 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/b341e7cd1bd6467bb439c9d2ee2c1b70 2023-04-17 08:01:24,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/3f83780cf001428999e11a75424f454b 2023-04-17 08:01:24,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/f803b65e222145c3b143d9bfc5918202 2023-04-17 08:01:24,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/48c741291a554ef8b7689405610d1ba7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/48c741291a554ef8b7689405610d1ba7 2023-04-17 08:01:24,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/2d356c4a188548d7a8fc70d52db56087 2023-04-17 08:01:24,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/66d671b3efc54946b669b361693c5bb4 2023-04-17 08:01:24,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/628b29fa22b748f78c8a980efccc0a4b 2023-04-17 08:01:24,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4cecc813bdd04ca4afa266899ad6a1cf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4cecc813bdd04ca4afa266899ad6a1cf 2023-04-17 08:01:24,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/e8723ab7e70c4ee0a36fda0760099c30 2023-04-17 08:01:24,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/156efd062d7f432ea4701b70a8029519 2023-04-17 08:01:24,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/8325e22a921d456394c3551728eb41a4 2023-04-17 08:01:24,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/recovered.edits/485.seqid, newMaxSeqId=485, maxSeqId=1 2023-04-17 08:01:24,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8. 2023-04-17 08:01:24,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for a8d6cbd896bae9327132746dceb281c8: 2023-04-17 08:01:24,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:24,661 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a8d6cbd896bae9327132746dceb281c8, regionState=CLOSED 2023-04-17 08:01:24,661 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718484661"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718484661"}]},"ts":"1681718484661"} 2023-04-17 08:01:24,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-17 08:01:24,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure a8d6cbd896bae9327132746dceb281c8, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.5520 sec 2023-04-17 08:01:24,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-17 08:01:24,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8d6cbd896bae9327132746dceb281c8, UNASSIGN in 1.5610 sec 2023-04-17 08:01:24,667 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718484667"}]},"ts":"1681718484667"} 2023-04-17 08:01:24,668 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:01:24,702 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:01:24,705 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6450 sec 2023-04-17 08:01:25,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 08:01:25,173 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-17 08:01:25,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:01:25,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,181 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,183 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 08:01:25,190 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:25,194 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/recovered.edits] 2023-04-17 08:01:25,198 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/17ad5e8fca42444394e3274ce0ed5828 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/17ad5e8fca42444394e3274ce0ed5828 2023-04-17 08:01:25,200 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b1a816a33c3481f9369a79087a5887e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/3b1a816a33c3481f9369a79087a5887e 2023-04-17 08:01:25,202 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4cb45fa19d1940959f922baced64db05 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/A/4cb45fa19d1940959f922baced64db05 2023-04-17 08:01:25,206 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4683d8faf6cf4fb1b889788a6f725b38 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/4683d8faf6cf4fb1b889788a6f725b38 2023-04-17 08:01:25,207 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/72256fc13ac54efc8b85274d3d5177fe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/72256fc13ac54efc8b85274d3d5177fe 2023-04-17 08:01:25,209 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c551e8e68b7e4df6991fa8429e5813b7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/B/c551e8e68b7e4df6991fa8429e5813b7 2023-04-17 08:01:25,212 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/1d59e126519f40e9978ce81d67c80490 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/1d59e126519f40e9978ce81d67c80490 2023-04-17 08:01:25,214 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4f4c3a4773334d4b8c6e18d9b87996a7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/4f4c3a4773334d4b8c6e18d9b87996a7 2023-04-17 08:01:25,216 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/edc65ec4ca754546ab3d67974b4b256a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/C/edc65ec4ca754546ab3d67974b4b256a 2023-04-17 08:01:25,219 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/recovered.edits/485.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8/recovered.edits/485.seqid 2023-04-17 08:01:25,220 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/a8d6cbd896bae9327132746dceb281c8 2023-04-17 08:01:25,220 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:01:25,221 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,223 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:01:25,224 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:01:25,225 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,225 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:01:25,225 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718485225"}]},"ts":"9223372036854775807"} 2023-04-17 08:01:25,226 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:01:25,226 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a8d6cbd896bae9327132746dceb281c8, NAME => 'TestAcidGuarantees,,1681718451645.a8d6cbd896bae9327132746dceb281c8.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:01:25,226 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:01:25,227 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718485227"}]},"ts":"9223372036854775807"} 2023-04-17 08:01:25,228 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:01:25,239 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-17 08:01:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 08:01:25,286 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-17 08:01:25,305 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=522 (was 495) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=319 (was 327), ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=7522 (was 7707) 2023-04-17 08:01:25,316 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=522, MaxFileDescriptor=60000, SystemLoadAverage=319, ProcessCount=168, AvailableMemoryMB=7522 2023-04-17 08:01:25,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:01:25,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:01:25,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:25,320 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 08:01:25,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-17 08:01:25,321 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 08:01:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 08:01:25,322 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:25,322 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 empty. 2023-04-17 08:01:25,323 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:25,323 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:01:25,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 08:01:25,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 08:01:25,731 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:01:25,734 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c74f57daa4f72232bba5b12448b01e11, NAME => 'TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 08:01:25,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:01:25,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c74f57daa4f72232bba5b12448b01e11, disabling compactions & flushes 2023-04-17 08:01:25,743 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:25,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:25,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. after waiting 0 ms 2023-04-17 08:01:25,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:25,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:25,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:25,747 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 08:01:25,748 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718485748"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718485748"}]},"ts":"1681718485748"} 2023-04-17 08:01:25,748 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 08:01:25,749 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 08:01:25,749 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718485749"}]},"ts":"1681718485749"} 2023-04-17 08:01:25,750 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 08:01:25,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, ASSIGN}] 2023-04-17 08:01:25,812 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, ASSIGN 2023-04-17 08:01:25,812 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 08:01:25,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 08:01:25,964 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:25,964 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718485964"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718485964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718485964"}]},"ts":"1681718485964"} 2023-04-17 08:01:25,965 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:01:26,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:26,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c74f57daa4f72232bba5b12448b01e11, NAME => 'TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:01:26,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:01:26,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,123 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,125 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A 2023-04-17 08:01:26,125 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A 2023-04-17 08:01:26,125 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:26,126 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName A 2023-04-17 08:01:26,126 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:26,127 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,128 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B 2023-04-17 08:01:26,129 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B 2023-04-17 08:01:26,129 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:26,129 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName B 2023-04-17 08:01:26,130 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:26,130 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,131 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C 2023-04-17 08:01:26,131 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C 2023-04-17 08:01:26,132 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:26,132 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName C 2023-04-17 08:01:26,133 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:26,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:26,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:01:26,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:26,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 08:01:26,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened c74f57daa4f72232bba5b12448b01e11; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69366190, jitterRate=0.033636778593063354}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:01:26,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:26,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., pid=37, masterSystemTime=1681718486118 2023-04-17 08:01:26,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:26,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:26,142 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:26,142 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718486142"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718486142"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718486142"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718486142"}]},"ts":"1681718486142"} 2023-04-17 08:01:26,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-17 08:01:26,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 in 178 msec 2023-04-17 08:01:26,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-17 08:01:26,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, ASSIGN in 334 msec 2023-04-17 08:01:26,146 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 08:01:26,146 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718486146"}]},"ts":"1681718486146"} 2023-04-17 08:01:26,147 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 08:01:26,181 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 08:01:26,182 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 863 msec 2023-04-17 08:01:26,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 08:01:26,427 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-17 08:01:26,428 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x377ba302 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:26,445 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d243e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:26,447 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:26,451 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:26,453 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 08:01:26,455 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 08:01:26,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:01:26,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:01:26,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:26,884 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:01:26,885 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:01:26,885 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:01:26,889 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 08:01:26,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, REOPEN/MOVE}] 2023-04-17 08:01:26,893 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, REOPEN/MOVE 2023-04-17 08:01:26,893 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:26,893 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718486893"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718486893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718486893"}]},"ts":"1681718486893"} 2023-04-17 08:01:26,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:01:27,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing c74f57daa4f72232bba5b12448b01e11, disabling compactions & flushes 2023-04-17 08:01:27,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. after waiting 0 ms 2023-04-17 08:01:27,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 08:01:27,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:27,055 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c74f57daa4f72232bba5b12448b01e11 to self. 2023-04-17 08:01:27,057 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,058 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=CLOSED 2023-04-17 08:01:27,058 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718487058"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718487058"}]},"ts":"1681718487058"} 2023-04-17 08:01:27,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-17 08:01:27,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 in 165 msec 2023-04-17 08:01:27,062 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=true 2023-04-17 08:01:27,213 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,213 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718487213"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718487213"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718487213"}]},"ts":"1681718487213"} 2023-04-17 08:01:27,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:01:27,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c74f57daa4f72232bba5b12448b01e11, NAME => 'TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:01:27,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:01:27,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,376 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,377 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A 2023-04-17 08:01:27,377 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A 2023-04-17 08:01:27,377 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:27,377 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName A 2023-04-17 08:01:27,379 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:27,379 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,380 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B 2023-04-17 08:01:27,380 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B 2023-04-17 08:01:27,380 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:27,380 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName B 2023-04-17 08:01:27,381 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:27,381 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,381 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C 2023-04-17 08:01:27,382 DEBUG [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C 2023-04-17 08:01:27,382 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:01:27,382 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c74f57daa4f72232bba5b12448b01e11 columnFamilyName C 2023-04-17 08:01:27,382 INFO [StoreOpener-c74f57daa4f72232bba5b12448b01e11-1] regionserver.HStore(310): Store=c74f57daa4f72232bba5b12448b01e11/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:01:27,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:01:27,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened c74f57daa4f72232bba5b12448b01e11; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64102883, jitterRate=-0.04479260742664337}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:01:27,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:27,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., pid=42, masterSystemTime=1681718487368 2023-04-17 08:01:27,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,391 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,391 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718487391"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718487391"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718487391"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718487391"}]},"ts":"1681718487391"} 2023-04-17 08:01:27,394 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-17 08:01:27,394 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 in 177 msec 2023-04-17 08:01:27,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-17 08:01:27,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, REOPEN/MOVE in 502 msec 2023-04-17 08:01:27,397 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-17 08:01:27,397 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 507 msec 2023-04-17 08:01:27,399 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 941 msec 2023-04-17 08:01:27,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-17 08:01:27,401 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x1eaa86ba to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,414 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a355de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,414 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x1d63a8ce to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,427 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57b6628e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,427 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5d80e5a8 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,439 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da5030e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,439 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x2ea1a2c1 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,452 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74d6bf18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,453 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x25185572 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,464 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60c5cba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,465 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x52249391 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,480 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a9ece3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,481 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5ba3ce53 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,494 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ded66ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,495 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5144fcff to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,510 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14d061d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,511 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4cc4 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,522 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@228ff1a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,523 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x1280c068 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:01:27,536 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@143a1290, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:01:27,544 DEBUG [hconnection-0x68d8126a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,544 DEBUG [hconnection-0x26bad705-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,545 DEBUG [hconnection-0x1f469a2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:27,547 DEBUG [hconnection-0x5c8d8505-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,548 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,548 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,549 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,549 DEBUG [hconnection-0x66a525ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,550 DEBUG [hconnection-0x1f9353-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,552 DEBUG [hconnection-0x6a46d337-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,552 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,552 DEBUG [hconnection-0x6b9a32fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,553 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:27,555 DEBUG [hconnection-0x2d192ed6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:27,559 DEBUG [hconnection-0x79f5cb6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:01:27,559 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:27,560 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,560 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:27,560 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:27,560 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,560 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,560 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,561 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,561 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,561 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,568 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:01:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:27,579 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:01:27,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:27,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:27,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:27,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:27,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:27,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:27,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718547594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718547594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,600 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:27,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718547594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718547595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718547595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:27,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:27,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:27,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:27,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:27,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:27,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:27,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:27,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:27,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603a3a6d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:27,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:27,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,615 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:27,622 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:27,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:27,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:27,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:27,632 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:27,632 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:27,632 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:27,632 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:27,632 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:27,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,640 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:27,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:27,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,647 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,647 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718547698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718547698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718547702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718547703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718547703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,747 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:27,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:27,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:27,750 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:27,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:27,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:27,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,750 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,755 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,755 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,756 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:27,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:27,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:27,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:27,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:27,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:27,757 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:27,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c03e218[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:27,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:27,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,772 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:27,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:27,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:27,782 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:27,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:27,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:27,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:27,782 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:27,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:27,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,797 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:27,797 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:27,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,897 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:27,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:27,900 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:27,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:27,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:27,901 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,901 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718547902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718547903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,905 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:27,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:27,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:27,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:27,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:27,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:27,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718547904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718547904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:27,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718547904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,913 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,914 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:27,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49688d8d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:27,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:27,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:27,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:27,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:27,932 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:27,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:27,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:27,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:27,932 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:27,938 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,938 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,938 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:27,939 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:27,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,947 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:27,947 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:27,947 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:27,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:27,990 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f8554fda3bd044d2a94f135d463c41e9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8554fda3bd044d2a94f135d463c41e9 2023-04-17 08:01:28,047 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,050 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,050 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,051 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,051 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,085 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,085 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,085 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,085 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c34bfb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,086 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,086 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,097 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,107 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,107 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,107 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,122 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,122 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718548204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718548206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718548208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718548208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718548210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,222 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,232 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,234 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,234 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,234 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,234 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,234 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,234 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,234 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,234 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,234 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129be4bc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,247 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,257 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,257 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,265 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,272 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,272 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,272 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,372 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,375 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,375 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,375 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,375 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,376 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,380 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,380 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,382 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,382 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,388 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cbd7d5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,395 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/1200004774fa451a8c9fb139f47ba494 2023-04-17 08:01:28,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,407 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,407 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,422 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,422 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,422 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,522 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,525 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,526 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,530 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,531 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,531 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,531 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,531 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,532 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c423c84[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,547 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,574 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,575 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,576 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,597 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,597 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,697 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,700 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718548707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718548708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65672864[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718548713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718548713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:28,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718548715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,722 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,732 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,732 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,738 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,738 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,738 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,747 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,747 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/9ccccb857f714a278252ed3d7d40d8a9 2023-04-17 08:01:28,847 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:28,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:28,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:28,851 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:28,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:28,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:28,852 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,852 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:28,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:28,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:28,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:28,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:28,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:28,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:28,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:28,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:28,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:28,857 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:28,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:28,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:28,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:28,857 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:28,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:28,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28149e70[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:28,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:28,865 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,872 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:28,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:28,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:28,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:28,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:28,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:28,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:28,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:28,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:28,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:28,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:28,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:28,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:28,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:28,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:28,897 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:28,897 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:28,897 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:28,997 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:28,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:29,000 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:29,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:29,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:29,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,000 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,005 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:29,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:29,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:29,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:29,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:29,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:29,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:29,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:29,006 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:29,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:29,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:29,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:29,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:29,007 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:29,007 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:29,007 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,007 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:29,007 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:29,013 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:29,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:29,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@276e81d8[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:29,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,022 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:29,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:29,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:29,022 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:29,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:29,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:29,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:29,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:29,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:29,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:29,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:29,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:29,058 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,059 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,059 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:29,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,061 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:29,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:29,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:29,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,079 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:29,079 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:29,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:29,180 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:29,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:29,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:29,182 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:29,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:29,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:29,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,183 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,188 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:29,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:29,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:29,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:29,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:29,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:29,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:29,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:29,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:29,191 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:29,206 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,206 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,206 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,206 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:29,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a052fad[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:29,208 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:29,208 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:29,214 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:29,214 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:29,214 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:29,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7cd9e683359e4ca29d1ad7993765d381 2023-04-17 08:01:29,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:29,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:29,222 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/1200004774fa451a8c9fb139f47ba494 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494 2023-04-17 08:01:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:29,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:29,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:29,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:29,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:29,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:29,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:29,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494, entries=250, sequenceid=16, filesize=38.7 K 2023-04-17 08:01:29,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/9ccccb857f714a278252ed3d7d40d8a9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9 2023-04-17 08:01:29,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:29,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:29,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:29,231 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:29,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9, entries=150, sequenceid=16, filesize=11.6 K 2023-04-17 08:01:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7cd9e683359e4ca29d1ad7993765d381 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381 2023-04-17 08:01:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:29,232 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381, entries=150, sequenceid=16, filesize=11.6 K 2023-04-17 08:01:29,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:29,238 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:29,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:29,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:29,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:29,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:29,239 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:29,239 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:29,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:29,239 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:29,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c74f57daa4f72232bba5b12448b01e11 in 1666ms, sequenceid=16, compaction requested=false 2023-04-17 08:01:29,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:29,339 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:29,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:29,343 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:29,343 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,343 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:29,343 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:29,344 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,344 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,354 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:29,354 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:29,354 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,354 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:29,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:29,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:29,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:29,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:29,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:29,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:29,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:29,356 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:29,356 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:29,356 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:29,357 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:29,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:29,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718549722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718549722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718549723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718549724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718549725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,779 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304174e2df2fdc8b74d2681d4bb9f4ef9f9e4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e2df2fdc8b74d2681d4bb9f4ef9f9e4 2023-04-17 08:01:29,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718549827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718549827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718549828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:29,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718549828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718550031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718550032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718550032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718550032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,184 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/61bbcbd18abe41a7b5909f95625fdd61 2023-04-17 08:01:30,188 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 08:01:30,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718550336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718550336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718550337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718550337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,595 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/8651fd66d19e415ab145a2574ac6df94 2023-04-17 08:01:30,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718550840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718550841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718550842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:30,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:30,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718550843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,004 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/2802720d47d041c98e5572493a22e0ac 2023-04-17 08:01:31,007 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/61bbcbd18abe41a7b5909f95625fdd61 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61 2023-04-17 08:01:31,010 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61, entries=150, sequenceid=36, filesize=25.1 K 2023-04-17 08:01:31,011 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/8651fd66d19e415ab145a2574ac6df94 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94 2023-04-17 08:01:31,014 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 08:01:31,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/2802720d47d041c98e5572493a22e0ac as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac 2023-04-17 08:01:31,017 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 08:01:31,018 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c74f57daa4f72232bba5b12448b01e11 in 1662ms, sequenceid=36, compaction requested=false 2023-04-17 08:01:31,018 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:31,018 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:31,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:31,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:31,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:31,018 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:31,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:31,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@367d0e81[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:31,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:31,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:31,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:31,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:31,064 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:31,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:31,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:31,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:31,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:31,074 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:31,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:31,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:31,074 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:31,074 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:31,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:31,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:31,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:31,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:31,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:31,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:31,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,089 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,089 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:31,089 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,089 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:31,089 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:31,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:31,189 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:31,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:31,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:31,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:31,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:31,192 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:31,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:31,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:31,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:31,193 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,193 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:31,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,197 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:31,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:31,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,197 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:31,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:31,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:31,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:31,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:31,198 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:31,198 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:31,198 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:31,610 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179c3070dae134475cb46fc629f913474f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179c3070dae134475cb46fc629f913474f 2023-04-17 08:01:31,614 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/c2ae7b143aaa451f8e7b0420e4f0110d 2023-04-17 08:01:31,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:31,745 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:31,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718551804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718551845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718551850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718551854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718551854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:31,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:31,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718551910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,022 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/a476f7846e41493d93543a5e1b232b4d 2023-04-17 08:01:32,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:32,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718552113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:32,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718552418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,430 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/bbfc4c9a77034be0a75b1fa73df514fe 2023-04-17 08:01:32,434 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/c2ae7b143aaa451f8e7b0420e4f0110d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d 2023-04-17 08:01:32,437 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d, entries=150, sequenceid=48, filesize=25.1 K 2023-04-17 08:01:32,438 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/a476f7846e41493d93543a5e1b232b4d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d 2023-04-17 08:01:32,441 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:01:32,441 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/bbfc4c9a77034be0a75b1fa73df514fe as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe 2023-04-17 08:01:32,445 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:01:32,445 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c74f57daa4f72232bba5b12448b01e11 in 1247ms, sequenceid=48, compaction requested=true 2023-04-17 08:01:32,445 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:32,445 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:32,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:32,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:32,445 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:32,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:32,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:32,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:32,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a96f120[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:32,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:32,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:32,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:32,489 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:32,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:32,497 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:32,497 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:32,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,497 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:32,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:32,498 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:32,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:32,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:32,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:32,498 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:32,505 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,505 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,505 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:32,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,505 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:32,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:32,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:32,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:32,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:32,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:32,534 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:32,534 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:32,534 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:32,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:32,534 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:32,535 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,535 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:32,535 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,635 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:32,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:32,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:32,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:32,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:32,638 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:32,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:32,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:32,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:32,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:32,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:32,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:32,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:32,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:32,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:32,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:32,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:32,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:32,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:32,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:32,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:32,648 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:32,648 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:32,648 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:32,649 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:32,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:32,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:32,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:32,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718552963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,061 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304176e9653d51bfa499485f36f803aaa9982 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176e9653d51bfa499485f36f803aaa9982 2023-04-17 08:01:33,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718553066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718553271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,465 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/f4e1d8852372424f83fb39472a330554 2023-04-17 08:01:33,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718553576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718553852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,857 DEBUG [Thread-2241] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:33,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718553862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,866 DEBUG [Thread-2243] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:33,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718553864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,869 DEBUG [Thread-2249] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:33,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:33,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718553871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,880 DEBUG [Thread-2245] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:33,883 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/27ce01514b144cfea7f83fa9aaf75894 2023-04-17 08:01:33,900 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/81ef4368eb304037a5b271539f98054c 2023-04-17 08:01:33,904 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/f4e1d8852372424f83fb39472a330554 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554 2023-04-17 08:01:33,907 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554, entries=150, sequenceid=68, filesize=25.1 K 2023-04-17 08:01:33,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/27ce01514b144cfea7f83fa9aaf75894 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894 2023-04-17 08:01:33,910 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894, entries=150, sequenceid=68, filesize=11.6 K 2023-04-17 08:01:33,911 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/81ef4368eb304037a5b271539f98054c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c 2023-04-17 08:01:33,914 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c, entries=150, sequenceid=68, filesize=11.6 K 2023-04-17 08:01:33,914 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c74f57daa4f72232bba5b12448b01e11 in 1266ms, sequenceid=68, compaction requested=true 2023-04-17 08:01:33,914 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:33,914 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:33,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:33,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:33,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:33,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6925d7eb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:33,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:33,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:33,963 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,963 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:33,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:33,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:33,971 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:33,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:33,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:33,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:33,973 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:33,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:33,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:33,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:33,973 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:33,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:33,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,981 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:33,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:33,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:33,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:33,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:33,989 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:33,989 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:33,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:33,989 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:33,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:33,989 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:34,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:34,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:34,089 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,094 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,094 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,095 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417053dbda3f3e14bec9f6672d2f8e3c32b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417053dbda3f3e14bec9f6672d2f8e3c32b 2023-04-17 08:01:34,095 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,095 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,121 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,121 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,121 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,121 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a1ef525[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,147 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,147 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,147 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,148 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,149 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,149 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,155 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,155 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,155 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,156 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,164 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,164 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:34,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718554193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,264 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,266 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,273 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,273 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,273 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,273 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c9efe1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,297 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,299 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,299 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,299 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:34,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718554299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,306 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,331 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,331 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,331 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,331 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,431 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,434 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,434 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,435 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,435 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,447 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,447 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,447 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795b0fa9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,473 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,473 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,489 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,489 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,489 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,500 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/dc9439ec4fff4c7986a45bd074347e6d 2023-04-17 08:01:34,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:34,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718554505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,589 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,593 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,593 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,593 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,593 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,593 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,593 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,597 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,598 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0605e6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,623 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,623 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,623 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,630 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,631 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,638 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,639 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,639 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,639 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,639 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,739 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,742 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,742 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,742 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,742 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,742 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,742 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,748 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,748 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,748 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,748 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,749 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,755 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63ec68de[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,771 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,773 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,774 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,774 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,781 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,788 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,789 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,789 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,789 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:34,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718554807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,889 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:34,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:34,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:34,893 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:34,893 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,893 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:34,893 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:34,894 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,894 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:34,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:34,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:34,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:34,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:34,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:34,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:34,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/bcbcc53662c44f048ac3df9fb4e45764 2023-04-17 08:01:34,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/e801f50e83c74a55a26e5cb60dbb5937 2023-04-17 08:01:34,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/dc9439ec4fff4c7986a45bd074347e6d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d 2023-04-17 08:01:34,931 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d, entries=200, sequenceid=81, filesize=31.9 K 2023-04-17 08:01:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/bcbcc53662c44f048ac3df9fb4e45764 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764 2023-04-17 08:01:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,934 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,934 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:34,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:34,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@586135a5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:34,934 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:34,934 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764, entries=150, sequenceid=81, filesize=11.6 K 2023-04-17 08:01:34,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/e801f50e83c74a55a26e5cb60dbb5937 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937 2023-04-17 08:01:34,947 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,947 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,947 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:34,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937, entries=150, sequenceid=81, filesize=11.6 K 2023-04-17 08:01:34,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,948 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:34,948 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:34,948 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:34,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c74f57daa4f72232bba5b12448b01e11 in 866ms, sequenceid=81, compaction requested=true 2023-04-17 08:01:34,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:34,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:34,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:34,949 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:01:34,949 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:01:34,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 149554 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:01:34,951 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:34,951 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=58.0 K 2023-04-17 08:01:34,951 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:34,951 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=146.0 K 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 9ccccb857f714a278252ed3d7d40d8a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718487574 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1200004774fa451a8c9fb139f47ba494, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718487574 2023-04-17 08:01:34,951 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8651fd66d19e415ab145a2574ac6df94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718487587 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 61bbcbd18abe41a7b5909f95625fdd61, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718487587 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a476f7846e41493d93543a5e1b232b4d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718489721 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting c2ae7b143aaa451f8e7b0420e4f0110d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718489721 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 27ce01514b144cfea7f83fa9aaf75894, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681718491769 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f4e1d8852372424f83fb39472a330554, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681718491769 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bcbcc53662c44f048ac3df9fb4e45764, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:34,952 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting dc9439ec4fff4c7986a45bd074347e6d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:34,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:34,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,956 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:34,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:34,958 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:34,958 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:34,958 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:34,958 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:34,958 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:34,960 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:34,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:34,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:34,966 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#309 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:34,966 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,969 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:34,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:34,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:34,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:34,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:34,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:34,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:34,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:34,981 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:34,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:34,981 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:34,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:34,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:34,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,081 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:35,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:35,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:35,083 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:35,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:35,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:35,083 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:35,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:35,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:35,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:35,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:35,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:35,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:35,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:35,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:35,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:35,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:35,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:35,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:35,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:35,106 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:35,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:35,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:35,369 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/549109dd744c4c32a79caa5665fd5649 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/549109dd744c4c32a79caa5665fd5649 2023-04-17 08:01:35,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:35,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718555363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,373 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into 549109dd744c4c32a79caa5665fd5649(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:35,373 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:35,373 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=11, startTime=1681718494949; duration=0sec 2023-04-17 08:01:35,373 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:35,373 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:01:35,375 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:01:35,375 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:35,375 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:35,375 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=58.0 K 2023-04-17 08:01:35,375 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7cd9e683359e4ca29d1ad7993765d381, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718487574 2023-04-17 08:01:35,376 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2802720d47d041c98e5572493a22e0ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718487587 2023-04-17 08:01:35,376 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bbfc4c9a77034be0a75b1fa73df514fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718489721 2023-04-17 08:01:35,376 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 81ef4368eb304037a5b271539f98054c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681718491769 2023-04-17 08:01:35,376 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e801f50e83c74a55a26e5cb60dbb5937, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:35,381 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/87ba597f2b2741eb97e8ca9f11e7eb73 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/87ba597f2b2741eb97e8ca9f11e7eb73 2023-04-17 08:01:35,383 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#311 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:35,385 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into 87ba597f2b2741eb97e8ca9f11e7eb73(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:35,385 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:35,385 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=11, startTime=1681718494949; duration=0sec 2023-04-17 08:01:35,385 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:35,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:35,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718555471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,520 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ce5301a88a84417b85b6785366da32e2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ce5301a88a84417b85b6785366da32e2 2023-04-17 08:01:35,524 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/752b6a1197b94a308090ea4fd2ea8a74 2023-04-17 08:01:35,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:35,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718555676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,790 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/57eae96067504726b355ecbbdd521017 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/57eae96067504726b355ecbbdd521017 2023-04-17 08:01:35,793 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into 57eae96067504726b355ecbbdd521017(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:35,793 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:35,794 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=11, startTime=1681718494949; duration=0sec 2023-04-17 08:01:35,794 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:35,933 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/97687605c9b8407184f6bf9e2a426c6a 2023-04-17 08:01:35,942 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/562b3d0ea77e48cf877a79a458b7f7da 2023-04-17 08:01:35,945 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/752b6a1197b94a308090ea4fd2ea8a74 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74 2023-04-17 08:01:35,949 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74, entries=150, sequenceid=100, filesize=25.1 K 2023-04-17 08:01:35,950 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/97687605c9b8407184f6bf9e2a426c6a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a 2023-04-17 08:01:35,953 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a, entries=150, sequenceid=100, filesize=11.6 K 2023-04-17 08:01:35,954 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/562b3d0ea77e48cf877a79a458b7f7da as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da 2023-04-17 08:01:35,956 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da, entries=150, sequenceid=100, filesize=11.6 K 2023-04-17 08:01:35,957 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c74f57daa4f72232bba5b12448b01e11 in 851ms, sequenceid=100, compaction requested=false 2023-04-17 08:01:35,957 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:35,957 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:35,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:35,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:35,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:35,957 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:35,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:35,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79665f47[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:35,964 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:35,964 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,971 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:35,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:35,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:35,972 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:35,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:35,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:35,982 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:35,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:35,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:35,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:35,982 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:35,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:35,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:35,985 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:35,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:35,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:35,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,989 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:35,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:35,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:35,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:35,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:35,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:35,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:35,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:35,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:35,997 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:35,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:35,997 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:35,997 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:36,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718556050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,097 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,100 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,100 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:36,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,107 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,114 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77414cb8[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,123 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,131 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,131 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,132 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,140 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,147 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,148 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,152 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,152 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,156 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718556159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,252 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,255 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:36,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,265 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,265 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,265 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,265 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,265 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,273 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77eb7016[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,316 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,317 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,317 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,317 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,317 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,317 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,332 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,332 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,332 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,338 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,338 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,338 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,347 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,347 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,347 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,347 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:36,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:36,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718556364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,399 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417201b33d97ff24fed8a5f7d90615bab99 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417201b33d97ff24fed8a5f7d90615bab99 2023-04-17 08:01:36,447 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,450 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:36,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,456 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,456 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,456 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,456 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,456 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2676af53[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,465 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,482 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,482 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,497 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,497 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,498 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,597 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,600 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:36,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,606 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,607 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,607 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da150bd[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,615 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,622 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,630 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,630 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,633 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,633 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,633 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,638 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,646 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,646 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:36,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,647 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,648 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,648 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,648 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718556669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,748 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,751 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,760 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,760 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,761 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,771 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f8f53d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,773 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,780 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,803 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/322ccda82e0241bba1bf1699d155932b 2023-04-17 08:01:36,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/87a641c00dc54b62bd5868744b802b6b 2023-04-17 08:01:36,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,812 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,815 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,815 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,822 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,823 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:36,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,830 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,830 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,831 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,931 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:36,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:36,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:36,933 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:36,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:36,933 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:36,934 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,934 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,938 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:36,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:36,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:36,939 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:36,939 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:36,939 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:36,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:36,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:36,940 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,940 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:36,941 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:36,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:36,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cbf5cf4[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:36,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:36,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,955 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:36,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:36,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:36,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:36,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:36,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:36,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:36,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:36,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:36,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:36,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:36,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:36,971 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,971 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:36,972 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:36,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,973 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:36,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:36,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:36,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:36,980 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:36,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:36,980 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:36,980 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:36,981 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,081 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:37,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:37,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:37,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:37,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:37,084 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:37,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:37,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:37,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:37,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,084 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:37,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:37,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:37,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:37,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:37,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:37,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:37,090 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:37,090 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:37,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:37,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:37,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:37,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f206007[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:37,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:37,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,105 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:37,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:37,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:37,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:37,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:37,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:37,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:37,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:37,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:37,115 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:37,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:37,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:37,115 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:37,115 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:37,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:37,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:37,130 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:37,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,130 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:37,130 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:37,130 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,131 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:37,131 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:37,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37314 deadline: 1681718557176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/c5fa4ac80f2d4c09b1392b32921c6b84 2023-04-17 08:01:37,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/322ccda82e0241bba1bf1699d155932b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b 2023-04-17 08:01:37,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b, entries=200, sequenceid=117, filesize=31.9 K 2023-04-17 08:01:37,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/87a641c00dc54b62bd5868744b802b6b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b 2023-04-17 08:01:37,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 08:01:37,231 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:37,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/c5fa4ac80f2d4c09b1392b32921c6b84 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84 2023-04-17 08:01:37,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:37,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:37,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:37,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:37,234 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:37,234 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:37,234 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:37,234 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:37,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 08:01:37,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c74f57daa4f72232bba5b12448b01e11 in 1250ms, sequenceid=117, compaction requested=true 2023-04-17 08:01:37,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:37,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:37,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:37,235 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:37,235 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:37,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:37,236 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,236 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,236 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:37,236 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:37,236 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,236 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/549109dd744c4c32a79caa5665fd5649, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=35.0 K 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84350 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 549109dd744c4c32a79caa5665fd5649, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:37,237 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,237 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/87ba597f2b2741eb97e8ca9f11e7eb73, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=82.4 K 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 97687605c9b8407184f6bf9e2a426c6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681718494183 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 87ba597f2b2741eb97e8ca9f11e7eb73, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:37,237 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 87a641c00dc54b62bd5868744b802b6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:37,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 752b6a1197b94a308090ea4fd2ea8a74, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681718494183 2023-04-17 08:01:37,238 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 322ccda82e0241bba1bf1699d155932b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:37,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:37,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:37,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:37,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:37,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:37,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:37,250 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:37,250 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:37,250 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:37,252 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:37,252 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:37,252 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,252 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:37,252 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:01:37,253 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:37,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:37,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:37,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:37,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:37,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:37,254 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:37,254 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#317 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:37,271 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/bfcf3086301548d79053d30b7cb92170 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bfcf3086301548d79053d30b7cb92170 2023-04-17 08:01:37,275 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into bfcf3086301548d79053d30b7cb92170(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:37,275 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:37,275 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=13, startTime=1681718497235; duration=0sec 2023-04-17 08:01:37,275 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:37,275 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:37,276 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:37,276 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:37,276 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:37,276 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/57eae96067504726b355ecbbdd521017, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=35.0 K 2023-04-17 08:01:37,276 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 57eae96067504726b355ecbbdd521017, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681718492939 2023-04-17 08:01:37,280 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 562b3d0ea77e48cf877a79a458b7f7da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681718494183 2023-04-17 08:01:37,281 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c5fa4ac80f2d4c09b1392b32921c6b84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:37,285 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#320 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:37,684 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172905cff20bf94faa9e87b53c7e7a15d0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172905cff20bf94faa9e87b53c7e7a15d0 2023-04-17 08:01:37,701 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/b3316e11d6964ea1998a1cbee5305294 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/b3316e11d6964ea1998a1cbee5305294 2023-04-17 08:01:37,701 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/c4e479c3d0b14e9ba531e16ffc44393e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c4e479c3d0b14e9ba531e16ffc44393e 2023-04-17 08:01:37,705 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into b3316e11d6964ea1998a1cbee5305294(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:37,705 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:37,705 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=13, startTime=1681718497235; duration=0sec 2023-04-17 08:01:37,706 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:37,706 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into c4e479c3d0b14e9ba531e16ffc44393e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:37,706 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:37,706 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=13, startTime=1681718497235; duration=0sec 2023-04-17 08:01:37,706 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:37,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:37,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718557895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:37,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718557898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718557907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,912 DEBUG [Thread-2249] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:37,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:37,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718557913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:37,918 DEBUG [Thread-2245] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:38,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718557999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718558000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,093 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/74f743ccb0ed4423ac4ebcd87877cc1c 2023-04-17 08:01:38,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37314 deadline: 1681718558187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718558205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718558205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,501 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/fec4b9120b484808a670697ee02e2ab3 2023-04-17 08:01:38,509 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/29ff9d56aab84c0a89b3a331a6447d13 2023-04-17 08:01:38,512 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/74f743ccb0ed4423ac4ebcd87877cc1c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c 2023-04-17 08:01:38,514 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c, entries=150, sequenceid=135, filesize=25.2 K 2023-04-17 08:01:38,515 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/fec4b9120b484808a670697ee02e2ab3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3 2023-04-17 08:01:38,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718558510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:38,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718558510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,518 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3, entries=150, sequenceid=135, filesize=11.7 K 2023-04-17 08:01:38,519 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/29ff9d56aab84c0a89b3a331a6447d13 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13 2023-04-17 08:01:38,522 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13, entries=150, sequenceid=135, filesize=11.7 K 2023-04-17 08:01:38,522 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c74f57daa4f72232bba5b12448b01e11 in 1270ms, sequenceid=135, compaction requested=false 2023-04-17 08:01:38,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:38,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:38,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:38,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:38,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:38,523 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:38,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,530 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:38,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:38,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:38,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4863e9e6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:38,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:38,531 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:38,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:38,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:38,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:38,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:38,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:38,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:38,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:38,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:38,548 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:38,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:38,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:38,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:38,548 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:38,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:38,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:38,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:38,584 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:38,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:38,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:38,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:38,584 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,584 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:38,584 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:38,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:38,685 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:38,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:38,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:38,687 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:38,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:38,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:38,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:38,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:38,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,697 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:38,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:38,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:38,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:38,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:38,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:38,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:38,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:38,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:38,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:38,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:38,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:38,698 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:38,698 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:38,698 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:38,698 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:01:38,698 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:38,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:38,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:38,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:38,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:38,699 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:38,714 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417b46d5c78322241288c151c537b955bd9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b46d5c78322241288c151c537b955bd9 2023-04-17 08:01:39,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:39,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:39,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718559066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718559067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,122 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/63d37913ca9c45149818114c894b301a 2023-04-17 08:01:39,164 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/72dce5604d6f4d1db28a31cdfb420bf4 2023-04-17 08:01:39,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718559174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718559175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,194 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/35c56dc644984b03b96ed1b9db6c603d 2023-04-17 08:01:39,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/63d37913ca9c45149818114c894b301a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a 2023-04-17 08:01:39,201 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a, entries=150, sequenceid=152, filesize=25.3 K 2023-04-17 08:01:39,202 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/72dce5604d6f4d1db28a31cdfb420bf4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4 2023-04-17 08:01:39,206 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4, entries=150, sequenceid=152, filesize=11.8 K 2023-04-17 08:01:39,207 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/35c56dc644984b03b96ed1b9db6c603d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d 2023-04-17 08:01:39,222 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d, entries=150, sequenceid=152, filesize=11.8 K 2023-04-17 08:01:39,223 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c74f57daa4f72232bba5b12448b01e11 in 525ms, sequenceid=152, compaction requested=true 2023-04-17 08:01:39,223 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:39,223 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:39,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:39,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:39,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:39,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:39,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe5bcca[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:39,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:39,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:39,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:39,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:39,264 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:39,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:39,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:39,274 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:39,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:39,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:39,274 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:39,274 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:39,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,289 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,289 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,289 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,292 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:39,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:39,292 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:39,292 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:39,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:39,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:39,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,392 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:39,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:39,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:39,396 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417b3f35f9595a34ccc913801644b97eb11 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b3f35f9595a34ccc913801644b97eb11 2023-04-17 08:01:39,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:39,396 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:39,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:39,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:39,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:39,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:39,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:39,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:39,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:39,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:39,407 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:39,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:39,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597b1b3e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:39,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:39,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:39,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:39,422 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,422 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:39,422 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:39,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718559421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718559421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:39,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:39,432 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:39,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:39,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:39,432 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:39,432 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:39,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,439 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,440 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,447 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,447 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,447 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:39,447 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718559526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718559527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,547 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:39,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:39,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:39,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:39,550 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:39,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:39,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:39,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:39,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:39,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:39,557 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,557 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:39,557 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:39,557 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:39,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,563 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:39,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:39,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f7784e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:39,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:39,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:39,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:39,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:39,582 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:39,582 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:39,582 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:39,582 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:39,582 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:39,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,588 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:39,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,618 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,619 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,618 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,619 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:39,619 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,619 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,619 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,719 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:39,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:39,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:39,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:39,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:39,721 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:39,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:39,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:39,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718559736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718559736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,752 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,752 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:39,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:39,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:39,753 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,753 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:39,753 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:39,753 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:39,753 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:39,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:39,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:39,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:39,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:39,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220c83cf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:39,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:39,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:39,772 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:39,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:39,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:39,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:39,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:39,782 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:39,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:39,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:39,782 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:39,782 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:39,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,788 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:39,788 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:39,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:39,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,797 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:39,797 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:39,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:39,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:39,797 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,801 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/2f2d5891cb244fd5ae5dae571e4871bf 2023-04-17 08:01:39,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/d6bfdb0fb49f4cacbe49be893aac1e91 2023-04-17 08:01:39,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7ac80dc3a7e34317b02d7c3c45732607 2023-04-17 08:01:39,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/2f2d5891cb244fd5ae5dae571e4871bf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf 2023-04-17 08:01:39,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf, entries=200, sequenceid=171, filesize=32.1 K 2023-04-17 08:01:39,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/d6bfdb0fb49f4cacbe49be893aac1e91 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91 2023-04-17 08:01:39,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91, entries=150, sequenceid=171, filesize=11.8 K 2023-04-17 08:01:39,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7ac80dc3a7e34317b02d7c3c45732607 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607 2023-04-17 08:01:39,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607, entries=150, sequenceid=171, filesize=11.8 K 2023-04-17 08:01:39,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c74f57daa4f72232bba5b12448b01e11 in 442ms, sequenceid=171, compaction requested=true 2023-04-17 08:01:39,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:39,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:39,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:39,828 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:39,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:39,829 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:39,829 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110648 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48218 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:39,830 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,830 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,830 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bfcf3086301548d79053d30b7cb92170, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=47.1 K 2023-04-17 08:01:39,830 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/b3316e11d6964ea1998a1cbee5305294, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=108.1 K 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bfcf3086301548d79053d30b7cb92170, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b3316e11d6964ea1998a1cbee5305294, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fec4b9120b484808a670697ee02e2ab3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681718496020 2023-04-17 08:01:39,830 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 74f743ccb0ed4423ac4ebcd87877cc1c, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681718496020 2023-04-17 08:01:39,831 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 72dce5604d6f4d1db28a31cdfb420bf4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681718497893 2023-04-17 08:01:39,831 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 63d37913ca9c45149818114c894b301a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681718497893 2023-04-17 08:01:39,831 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d6bfdb0fb49f4cacbe49be893aac1e91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:39,831 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2f2d5891cb244fd5ae5dae571e4871bf, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:39,837 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#329 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:39,857 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#330 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:39,860 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/4389a1ca2a2442108d6c65f47f58e110 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4389a1ca2a2442108d6c65f47f58e110 2023-04-17 08:01:39,868 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into 4389a1ca2a2442108d6c65f47f58e110(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:39,868 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:39,868 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=12, startTime=1681718499828; duration=0sec 2023-04-17 08:01:39,868 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:39,868 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:39,869 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48218 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:39,869 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:39,869 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,869 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c4e479c3d0b14e9ba531e16ffc44393e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=47.1 K 2023-04-17 08:01:39,870 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c4e479c3d0b14e9ba531e16ffc44393e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681718495320 2023-04-17 08:01:39,870 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 29ff9d56aab84c0a89b3a331a6447d13, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681718496020 2023-04-17 08:01:39,871 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 35c56dc644984b03b96ed1b9db6c603d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681718497893 2023-04-17 08:01:39,871 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7ac80dc3a7e34317b02d7c3c45732607, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:39,882 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#331 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:39,896 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/addcd3692c2440e2943607dc3af29a15 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/addcd3692c2440e2943607dc3af29a15 2023-04-17 08:01:39,897 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:39,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:39,898 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/5c84edb3b2d5489e83fb6e1fa9b064c2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5c84edb3b2d5489e83fb6e1fa9b064c2 2023-04-17 08:01:39,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:39,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:39,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:39,900 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:39,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:39,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:39,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,900 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,902 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into addcd3692c2440e2943607dc3af29a15(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:39,902 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:39,902 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=12, startTime=1681718499829; duration=0sec 2023-04-17 08:01:39,902 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:39,904 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into 5c84edb3b2d5489e83fb6e1fa9b064c2(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:39,904 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:39,904 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=12, startTime=1681718499828; duration=0sec 2023-04-17 08:01:39,904 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:39,905 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:39,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:39,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:39,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:39,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:39,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:39,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:39,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:39,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:39,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:39,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:39,906 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:39,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:39,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:39,906 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:01:39,906 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:39,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:39,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:39,907 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:39,915 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a9940fe3057e4403aff3aa2c3585bec9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a9940fe3057e4403aff3aa2c3585bec9 2023-04-17 08:01:39,919 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=188, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/9d23fbf5f70d40698400791c93a705b4 2023-04-17 08:01:40,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:40,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:40,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718560071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718560072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718560174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718560176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37314 deadline: 1681718560201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,204 DEBUG [Thread-2247] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:40,327 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/1c3342b946e04c12b9cc1556644db973 2023-04-17 08:01:40,337 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/dad50fb09ca44491a2b9e62a0042b8af 2023-04-17 08:01:40,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/9d23fbf5f70d40698400791c93a705b4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4 2023-04-17 08:01:40,342 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4, entries=150, sequenceid=188, filesize=25.3 K 2023-04-17 08:01:40,343 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/1c3342b946e04c12b9cc1556644db973 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973 2023-04-17 08:01:40,345 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973, entries=150, sequenceid=188, filesize=11.8 K 2023-04-17 08:01:40,346 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/dad50fb09ca44491a2b9e62a0042b8af as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af 2023-04-17 08:01:40,350 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af, entries=150, sequenceid=188, filesize=11.8 K 2023-04-17 08:01:40,352 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c74f57daa4f72232bba5b12448b01e11 in 446ms, sequenceid=188, compaction requested=false 2023-04-17 08:01:40,352 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:40,353 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:40,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:40,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:40,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:40,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:40,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103d78a8[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:40,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:40,401 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,405 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:40,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:40,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:40,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:40,412 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041796394aa0bb1c43fdbca21236256a4d4a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796394aa0bb1c43fdbca21236256a4d4a 2023-04-17 08:01:40,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:40,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:40,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:40,415 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:40,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:40,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:40,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:40,416 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:40,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,423 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,422 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718560421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718560424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:40,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,430 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,430 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,431 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718560526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718560528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,530 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:40,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:40,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:40,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:40,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:40,533 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:40,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:40,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:40,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:40,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:40,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:40,540 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,540 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:40,540 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:40,540 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:40,541 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:40,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:40,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119e237a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:40,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:40,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:40,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:40,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:40,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:40,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:40,563 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:40,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:40,565 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:40,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:40,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:40,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:40,565 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:40,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:40,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,580 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:40,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,580 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:40,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,581 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,680 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:40,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:40,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:40,683 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:40,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:40,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:40,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,688 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,688 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:40,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:40,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:40,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:40,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:40,690 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,690 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:40,690 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:40,690 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:40,690 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:40,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,697 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,697 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:40,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:40,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f650c0f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:40,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:40,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:40,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:40,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:40,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:40,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,713 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:40,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:40,715 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:40,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:40,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:40,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:40,715 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:40,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:40,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,730 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:40,730 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:40,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718560730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:40,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718560732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,821 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/534603683c23406fbacdab9cccd2b853 2023-04-17 08:01:40,830 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:40,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:40,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/00ce4c58efd64898a0b4b4cfa6c9bd35 2023-04-17 08:01:40,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:40,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:40,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:40,834 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:40,834 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,834 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:40,834 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:40,835 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,835 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:40,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:40,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:40,848 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,848 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:40,848 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:40,848 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:40,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:40,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:40,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b1eed7[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:40,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:40,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:40,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:40,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:40,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:40,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:40,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:40,874 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:40,875 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:40,875 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:40,875 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:40,875 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:40,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:40,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:40,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,881 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:40,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:40,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:40,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:40,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,889 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:40,889 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:40,889 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:40,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:40,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:40,989 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:40,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:40,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:40,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:40,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:40,992 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:40,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:40,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:40,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,992 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,996 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:40,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:40,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:40,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:40,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:40,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:40,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:40,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:40,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:40,998 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:40,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:40,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:40,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,000 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:41,000 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:41,000 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,000 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:41,000 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:41,000 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,000 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:41,000 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:41,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,005 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:41,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62323d71[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:41,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:41,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,013 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:41,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:41,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:41,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:41,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:41,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:41,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:41,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:41,023 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:41,023 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:41,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:41,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,030 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718561034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:41,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:41,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:41,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:41,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:41,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718561037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,039 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:41,039 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:41,039 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:41,139 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:41,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:41,142 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:41,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:41,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:41,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:41,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,142 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,146 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:41,146 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:41,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:41,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:41,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:41,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:41,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:41,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:41,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:41,159 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:41,160 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,160 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:41,160 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:41,160 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:41,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:41,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:41,160 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:41,171 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:41,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d757be[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:41,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,200 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:41,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:41,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:41,200 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:41,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:41,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:41,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:41,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:41,215 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:41,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:41,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:41,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:41,216 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:41,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:41,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:41,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,223 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:41,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:41,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:41,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:41,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:41,230 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:41,230 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:41,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/59838adde4cc48908f686150f009c8c5 2023-04-17 08:01:41,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/534603683c23406fbacdab9cccd2b853 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853 2023-04-17 08:01:41,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853, entries=200, sequenceid=207, filesize=32.1 K 2023-04-17 08:01:41,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/00ce4c58efd64898a0b4b4cfa6c9bd35 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35 2023-04-17 08:01:41,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35, entries=150, sequenceid=207, filesize=11.8 K 2023-04-17 08:01:41,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/59838adde4cc48908f686150f009c8c5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5 2023-04-17 08:01:41,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5, entries=150, sequenceid=207, filesize=11.8 K 2023-04-17 08:01:41,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c74f57daa4f72232bba5b12448b01e11 in 877ms, sequenceid=207, compaction requested=true 2023-04-17 08:01:41,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:41,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:41,258 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:41,258 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:41,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:41,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:41,259 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:41,259 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:41,259 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:41,259 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:41,259 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,259 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,259 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5c84edb3b2d5489e83fb6e1fa9b064c2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=83.1 K 2023-04-17 08:01:41,259 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4389a1ca2a2442108d6c65f47f58e110, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=35.7 K 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5c84edb3b2d5489e83fb6e1fa9b064c2, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4389a1ca2a2442108d6c65f47f58e110, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9d23fbf5f70d40698400791c93a705b4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681718499413 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1c3342b946e04c12b9cc1556644db973, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681718499413 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 534603683c23406fbacdab9cccd2b853, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500065 2023-04-17 08:01:41,260 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 00ce4c58efd64898a0b4b4cfa6c9bd35, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500068 2023-04-17 08:01:41,269 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:41,277 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#338 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:41,285 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/495d9454d0604ff9b1ce58debed3f90f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/495d9454d0604ff9b1ce58debed3f90f 2023-04-17 08:01:41,290 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into 495d9454d0604ff9b1ce58debed3f90f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:41,291 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:41,291 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=13, startTime=1681718501258; duration=0sec 2023-04-17 08:01:41,291 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:41,291 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:01:41,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:01:41,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:41,292 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,292 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/addcd3692c2440e2943607dc3af29a15, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=35.7 K 2023-04-17 08:01:41,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting addcd3692c2440e2943607dc3af29a15, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681718499025 2023-04-17 08:01:41,292 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting dad50fb09ca44491a2b9e62a0042b8af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681718499413 2023-04-17 08:01:41,293 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 59838adde4cc48908f686150f009c8c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500068 2023-04-17 08:01:41,301 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#340 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:41,312 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/eb1a0bb980574b3099a861fa98100338 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb1a0bb980574b3099a861fa98100338 2023-04-17 08:01:41,316 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into eb1a0bb980574b3099a861fa98100338(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:41,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:41,316 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=13, startTime=1681718501258; duration=0sec 2023-04-17 08:01:41,317 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:41,331 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:41,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:41,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:41,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:41,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:41,334 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:41,334 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:41,334 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:41,334 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:41,335 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,336 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:41,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:41,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:41,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:41,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:41,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:41,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:41,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:41,349 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:41,358 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:41,358 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:41,358 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:41,358 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:41,358 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:41,359 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:41,375 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417cf7ac500f3724ad6a56c75b6b81064ea to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cf7ac500f3724ad6a56c75b6b81064ea 2023-04-17 08:01:41,401 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=222, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/cb867e4975aa4fcf9dd0197cc35ce4c2 2023-04-17 08:01:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:41,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:41,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718561588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718561589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718561701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718561703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,819 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/2144f897807747ebaf087d534dd0eb70 2023-04-17 08:01:41,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718561903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:41,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:41,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718561904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718562207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718562207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,229 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7c3ee940bca14eceb8886ea8a50e2cf7 2023-04-17 08:01:42,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/cb867e4975aa4fcf9dd0197cc35ce4c2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2 2023-04-17 08:01:42,235 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2, entries=150, sequenceid=222, filesize=25.3 K 2023-04-17 08:01:42,236 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/2144f897807747ebaf087d534dd0eb70 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70 2023-04-17 08:01:42,239 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70, entries=150, sequenceid=222, filesize=11.8 K 2023-04-17 08:01:42,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/7c3ee940bca14eceb8886ea8a50e2cf7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7 2023-04-17 08:01:42,243 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7, entries=150, sequenceid=222, filesize=11.8 K 2023-04-17 08:01:42,243 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c74f57daa4f72232bba5b12448b01e11 in 885ms, sequenceid=222, compaction requested=false 2023-04-17 08:01:42,243 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:42,243 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:42,243 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:42,243 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:42,243 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,243 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:42,244 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:42,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:42,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,272 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:42,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:42,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a5369f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:42,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:42,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,280 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:42,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:42,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:42,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:42,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:42,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:42,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:42,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:42,290 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:42,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:42,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:42,290 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:42,290 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:42,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:42,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:42,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:42,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:42,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:42,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:42,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:42,305 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:42,305 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:42,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,405 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:42,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:42,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:42,408 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:42,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:42,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:42,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:42,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:42,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:42,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:42,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:42,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:42,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:42,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:42,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:42,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:42,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:42,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:42,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:42,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:42,421 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:42,421 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:42,516 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/7e5befe8450c4436bc813009a3883e85 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/7e5befe8450c4436bc813009a3883e85 2023-04-17 08:01:42,519 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into 7e5befe8450c4436bc813009a3883e85(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:01:42,519 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:42,519 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=13, startTime=1681718501258; duration=1sec 2023-04-17 08:01:42,519 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:42,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:42,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718562731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718562732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,833 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041799bc20718ce7420aa2e6ed43fac49e88 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041799bc20718ce7420aa2e6ed43fac49e88 2023-04-17 08:01:42,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718562836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:42,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:42,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718562837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718563043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718563043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,242 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/9b21719814ff4dce9e0bf15a0be087a6 2023-04-17 08:01:43,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37364 deadline: 1681718563347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37388 deadline: 1681718563350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,650 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/03a8ce12edad4d22a19aebc87feb50af 2023-04-17 08:01:43,658 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/2db06bf43e2a4269b52f5be26a84aaef 2023-04-17 08:01:43,662 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/9b21719814ff4dce9e0bf15a0be087a6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6 2023-04-17 08:01:43,665 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6, entries=150, sequenceid=241, filesize=25.3 K 2023-04-17 08:01:43,666 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/03a8ce12edad4d22a19aebc87feb50af as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af 2023-04-17 08:01:43,675 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af, entries=150, sequenceid=241, filesize=11.8 K 2023-04-17 08:01:43,675 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/2db06bf43e2a4269b52f5be26a84aaef as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef 2023-04-17 08:01:43,688 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef, entries=150, sequenceid=241, filesize=11.8 K 2023-04-17 08:01:43,689 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c74f57daa4f72232bba5b12448b01e11 in 1268ms, sequenceid=241, compaction requested=true 2023-04-17 08:01:43,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:43,689 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:43,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:43,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:43,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:43,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:43,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,721 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:43,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6374b9aa[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:43,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:43,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,730 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:43,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:43,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:43,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:43,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:43,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:43,738 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:43,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:43,740 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:43,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:43,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:43,740 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:43,740 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:43,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:43,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:43,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:43,755 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:43,755 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:43,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:43,755 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,755 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,855 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:43,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:01:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:43,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:43,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:43,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:43,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:43,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:43,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:43,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:43,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:43,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,863 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:43,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:43,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:43,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:43,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:43,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:43,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:43,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:43,864 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:43,864 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:43,864 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:43,865 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:43,865 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:43,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:43,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:43,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:43,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:43,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,872 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:43,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:43,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10dec900[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:43,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,880 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:43,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:43,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:43,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:43,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:43,905 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:43,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:43,907 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:43,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:43,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:43,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:43,907 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:43,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718563907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:43,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718563908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:43,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:43,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:43,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:43,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:43,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:43,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:43,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:43,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:43,930 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:43,930 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:43,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718564014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718564014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,030 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,033 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288c1602[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,065 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,065 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,065 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:44,080 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,080 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,080 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,180 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,184 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,184 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,188 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,190 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,190 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,196 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7441dc42[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,198 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,205 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,205 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,205 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,205 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,213 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,215 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,215 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718564220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718564220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,230 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,230 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:44,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,230 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37314 deadline: 1681718564241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,247 DEBUG [Thread-2247] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:44,265 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041723219b44a44442f681aa2d306d2e039a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041723219b44a44442f681aa2d306d2e039a 2023-04-17 08:01:44,330 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,333 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,333 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,338 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,339 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,340 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,340 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,340 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,340 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32be8289[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,348 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,355 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,355 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,363 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,363 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,363 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,363 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,365 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,365 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,365 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,372 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,397 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,398 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,398 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,398 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,498 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,501 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,501 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,513 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,513 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,513 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,513 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,515 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,515 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,515 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,515 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,515 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544edb17[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,530 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718564527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718564527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,540 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,540 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,540 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,555 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,555 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,555 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:44,655 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,658 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,658 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,663 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,663 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,663 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,663 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,664 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,665 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,665 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,670 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/028a689a70e94c84a49bcee3a473cc3d 2023-04-17 08:01:44,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,671 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbaeb9a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,690 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,690 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:44,705 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,705 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,805 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,808 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,808 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,813 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,813 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,813 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,813 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,814 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,814 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,815 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,815 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,821 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,823 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66639783[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,823 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,830 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:44,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:44,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:44,830 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:44,838 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:44,838 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,838 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:44,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:44,840 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:44,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:44,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:44,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:44,840 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:44,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,847 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,848 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:44,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:44,873 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,873 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,873 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,973 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:44,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:44,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:44,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:44,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:44,976 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:44,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:44,976 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:44,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:44,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:44,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:44,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:44,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:44,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:44,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:44,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:44,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:44,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:44,989 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:44,989 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,989 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:44,990 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:44,990 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:44,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:44,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:44,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:44,990 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:44,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:44,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:44,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:44,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:44,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:44,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200feb0c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:44,998 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,005 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:45,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:45,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:45,005 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:45,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:45,013 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:45,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:45,015 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:45,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:45,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:45,015 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:45,015 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:45,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,030 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:45,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,030 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:45,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:45,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,030 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:45,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:45,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718565031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:45,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718565032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/36e5f4aa6b284f4db7d2d29193d54ee5 2023-04-17 08:01:45,131 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:45,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:45,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:45,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:45,133 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:45,133 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,133 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:45,133 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:45,134 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,134 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:45,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:45,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:45,139 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,139 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:45,139 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:45,139 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:45,139 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:45,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:45,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:45,146 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,146 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,147 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,147 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:45,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:45,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb1a79[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:45,148 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,155 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:45,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:45,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:45,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:45,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:45,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,163 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,163 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:45,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:45,165 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:45,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:45,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:45,165 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:45,165 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:45,171 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,171 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,171 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:45,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,180 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,180 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:45,180 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:45,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,280 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:45,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:45,283 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:45,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:45,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:45,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,283 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:45,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:45,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:45,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:45,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:45,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:45,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:45,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:45,290 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:45,290 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:45,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:45,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:45,290 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:45,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,297 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:45,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4916c78f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:45,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:45,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:45,305 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:45,305 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:45,305 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:45,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:45,313 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:45,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:45,315 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:45,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:45,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:45,315 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:45,315 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:45,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,322 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:45,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:45,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,330 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:45,330 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:45,330 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:45,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,430 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:45,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:45,433 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:45,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:45,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:45,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,433 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,457 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,457 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:45,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:45,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:45,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:45,458 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,458 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:45,458 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:45,458 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:45,458 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:45,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,472 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:45,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33339ea1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:45,473 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,480 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:45,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:45,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:45,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:45,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:45,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:45,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:45,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:45,490 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:45,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:45,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:45,490 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:45,490 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:45,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/1a8c2762a90147a28a07038be850fa2f 2023-04-17 08:01:45,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/028a689a70e94c84a49bcee3a473cc3d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d 2023-04-17 08:01:45,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:45,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,497 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,498 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:45,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d, entries=200, sequenceid=256, filesize=32.1 K 2023-04-17 08:01:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/36e5f4aa6b284f4db7d2d29193d54ee5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5 2023-04-17 08:01:45,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5, entries=150, sequenceid=256, filesize=11.8 K 2023-04-17 08:01:45,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/1a8c2762a90147a28a07038be850fa2f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f 2023-04-17 08:01:45,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f, entries=150, sequenceid=256, filesize=11.8 K 2023-04-17 08:01:45,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for c74f57daa4f72232bba5b12448b01e11 in 1652ms, sequenceid=256, compaction requested=true 2023-04-17 08:01:45,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:45,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:45,507 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:45,507 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:45,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:45,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:45,508 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:45,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:45,508 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:45,509 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:45,509 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,509 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,509 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/7e5befe8450c4436bc813009a3883e85, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=108.5 K 2023-04-17 08:01:45,509 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/495d9454d0604ff9b1ce58debed3f90f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=47.5 K 2023-04-17 08:01:45,509 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 495d9454d0604ff9b1ce58debed3f90f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500068 2023-04-17 08:01:45,509 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7e5befe8450c4436bc813009a3883e85, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500068 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2144f897807747ebaf087d534dd0eb70, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681718500419 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting cb867e4975aa4fcf9dd0197cc35ce4c2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681718500419 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 03a8ce12edad4d22a19aebc87feb50af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681718501581 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9b21719814ff4dce9e0bf15a0be087a6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681718501581 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 36e5f4aa6b284f4db7d2d29193d54ee5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502730 2023-04-17 08:01:45,510 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 028a689a70e94c84a49bcee3a473cc3d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502720 2023-04-17 08:01:45,518 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:45,522 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#350 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:45,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:45,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:45,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:45,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,591 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:45,592 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,592 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,592 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:45,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,592 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:45,592 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,592 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:45,592 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:45,692 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:45,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:45,694 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:45,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:45,694 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:45,695 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,695 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,705 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:45,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:45,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:45,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:45,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:45,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:45,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:45,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:45,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:45,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:45,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:45,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:45,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:45,707 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:45,707 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:45,927 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/3692253d7bcd40ea9365fad4d8ddd31e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/3692253d7bcd40ea9365fad4d8ddd31e 2023-04-17 08:01:45,930 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into 3692253d7bcd40ea9365fad4d8ddd31e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:45,931 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:45,931 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=12, startTime=1681718505507; duration=0sec 2023-04-17 08:01:45,931 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:45,931 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:45,932 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:45,932 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:45,933 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:45,933 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb1a0bb980574b3099a861fa98100338, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=47.5 K 2023-04-17 08:01:45,933 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting eb1a0bb980574b3099a861fa98100338, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681718500068 2023-04-17 08:01:45,933 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7c3ee940bca14eceb8886ea8a50e2cf7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681718500419 2023-04-17 08:01:45,934 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2db06bf43e2a4269b52f5be26a84aaef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681718501581 2023-04-17 08:01:45,934 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1a8c2762a90147a28a07038be850fa2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502730 2023-04-17 08:01:45,937 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/5cdbccefedde427782a97e5a52426ee6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5cdbccefedde427782a97e5a52426ee6 2023-04-17 08:01:45,940 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#353 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 08:01:45,940 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into 5cdbccefedde427782a97e5a52426ee6(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:45,940 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:45,940 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=12, startTime=1681718505507; duration=0sec 2023-04-17 08:01:45,941 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:46,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:46,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:46,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718566059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718566060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,126 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304175838cdd221e942cdb2f97eb6871635d5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175838cdd221e942cdb2f97eb6871635d5 2023-04-17 08:01:46,130 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/3319610d0f654217b0933c7d941efabd 2023-04-17 08:01:46,165 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/87ba597f2b2741eb97e8ca9f11e7eb73, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/b3316e11d6964ea1998a1cbee5305294, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5c84edb3b2d5489e83fb6e1fa9b064c2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/7e5befe8450c4436bc813009a3883e85, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d] to archive 2023-04-17 08:01:46,166 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:46,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718566165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,167 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/1200004774fa451a8c9fb139f47ba494 2023-04-17 08:01:46,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718566166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,169 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/61bbcbd18abe41a7b5909f95625fdd61 2023-04-17 08:01:46,171 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/c2ae7b143aaa451f8e7b0420e4f0110d 2023-04-17 08:01:46,172 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/f4e1d8852372424f83fb39472a330554 2023-04-17 08:01:46,173 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/dc9439ec4fff4c7986a45bd074347e6d 2023-04-17 08:01:46,174 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/87ba597f2b2741eb97e8ca9f11e7eb73 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/87ba597f2b2741eb97e8ca9f11e7eb73 2023-04-17 08:01:46,175 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/752b6a1197b94a308090ea4fd2ea8a74 2023-04-17 08:01:46,176 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/322ccda82e0241bba1bf1699d155932b 2023-04-17 08:01:46,177 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/b3316e11d6964ea1998a1cbee5305294 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/b3316e11d6964ea1998a1cbee5305294 2023-04-17 08:01:46,178 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/74f743ccb0ed4423ac4ebcd87877cc1c 2023-04-17 08:01:46,179 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/63d37913ca9c45149818114c894b301a 2023-04-17 08:01:46,180 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2f2d5891cb244fd5ae5dae571e4871bf 2023-04-17 08:01:46,181 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5c84edb3b2d5489e83fb6e1fa9b064c2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5c84edb3b2d5489e83fb6e1fa9b064c2 2023-04-17 08:01:46,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9d23fbf5f70d40698400791c93a705b4 2023-04-17 08:01:46,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/534603683c23406fbacdab9cccd2b853 2023-04-17 08:01:46,183 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/7e5befe8450c4436bc813009a3883e85 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/7e5befe8450c4436bc813009a3883e85 2023-04-17 08:01:46,184 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cb867e4975aa4fcf9dd0197cc35ce4c2 2023-04-17 08:01:46,184 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/9b21719814ff4dce9e0bf15a0be087a6 2023-04-17 08:01:46,185 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/028a689a70e94c84a49bcee3a473cc3d 2023-04-17 08:01:46,194 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/549109dd744c4c32a79caa5665fd5649, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bfcf3086301548d79053d30b7cb92170, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4389a1ca2a2442108d6c65f47f58e110, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/495d9454d0604ff9b1ce58debed3f90f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5] to archive 2023-04-17 08:01:46,195 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:46,196 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/9ccccb857f714a278252ed3d7d40d8a9 2023-04-17 08:01:46,197 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/8651fd66d19e415ab145a2574ac6df94 2023-04-17 08:01:46,198 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/a476f7846e41493d93543a5e1b232b4d 2023-04-17 08:01:46,199 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/27ce01514b144cfea7f83fa9aaf75894 2023-04-17 08:01:46,199 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/549109dd744c4c32a79caa5665fd5649 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/549109dd744c4c32a79caa5665fd5649 2023-04-17 08:01:46,200 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bcbcc53662c44f048ac3df9fb4e45764 2023-04-17 08:01:46,201 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/97687605c9b8407184f6bf9e2a426c6a 2023-04-17 08:01:46,202 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bfcf3086301548d79053d30b7cb92170 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/bfcf3086301548d79053d30b7cb92170 2023-04-17 08:01:46,203 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/87a641c00dc54b62bd5868744b802b6b 2023-04-17 08:01:46,204 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/fec4b9120b484808a670697ee02e2ab3 2023-04-17 08:01:46,205 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/72dce5604d6f4d1db28a31cdfb420bf4 2023-04-17 08:01:46,206 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4389a1ca2a2442108d6c65f47f58e110 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4389a1ca2a2442108d6c65f47f58e110 2023-04-17 08:01:46,206 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/d6bfdb0fb49f4cacbe49be893aac1e91 2023-04-17 08:01:46,207 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/1c3342b946e04c12b9cc1556644db973 2023-04-17 08:01:46,208 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/495d9454d0604ff9b1ce58debed3f90f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/495d9454d0604ff9b1ce58debed3f90f 2023-04-17 08:01:46,208 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/00ce4c58efd64898a0b4b4cfa6c9bd35 2023-04-17 08:01:46,209 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/2144f897807747ebaf087d534dd0eb70 2023-04-17 08:01:46,210 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/03a8ce12edad4d22a19aebc87feb50af 2023-04-17 08:01:46,210 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/36e5f4aa6b284f4db7d2d29193d54ee5 2023-04-17 08:01:46,213 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/57eae96067504726b355ecbbdd521017, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c4e479c3d0b14e9ba531e16ffc44393e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/addcd3692c2440e2943607dc3af29a15, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5] to archive 2023-04-17 08:01:46,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:46,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7cd9e683359e4ca29d1ad7993765d381 2023-04-17 08:01:46,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2802720d47d041c98e5572493a22e0ac 2023-04-17 08:01:46,216 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/bbfc4c9a77034be0a75b1fa73df514fe 2023-04-17 08:01:46,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/81ef4368eb304037a5b271539f98054c 2023-04-17 08:01:46,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/57eae96067504726b355ecbbdd521017 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/57eae96067504726b355ecbbdd521017 2023-04-17 08:01:46,218 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e801f50e83c74a55a26e5cb60dbb5937 2023-04-17 08:01:46,219 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/562b3d0ea77e48cf877a79a458b7f7da 2023-04-17 08:01:46,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c4e479c3d0b14e9ba531e16ffc44393e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c4e479c3d0b14e9ba531e16ffc44393e 2023-04-17 08:01:46,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/c5fa4ac80f2d4c09b1392b32921c6b84 2023-04-17 08:01:46,222 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/29ff9d56aab84c0a89b3a331a6447d13 2023-04-17 08:01:46,222 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/35c56dc644984b03b96ed1b9db6c603d 2023-04-17 08:01:46,223 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/addcd3692c2440e2943607dc3af29a15 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/addcd3692c2440e2943607dc3af29a15 2023-04-17 08:01:46,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7ac80dc3a7e34317b02d7c3c45732607 2023-04-17 08:01:46,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/dad50fb09ca44491a2b9e62a0042b8af 2023-04-17 08:01:46,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/59838adde4cc48908f686150f009c8c5 2023-04-17 08:01:46,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/16e9b8b393ef4aa0b354ea927a8dc3d6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/16e9b8b393ef4aa0b354ea927a8dc3d6 2023-04-17 08:01:46,352 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into 16e9b8b393ef4aa0b354ea927a8dc3d6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:46,353 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:46,353 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=12, startTime=1681718505508; duration=0sec 2023-04-17 08:01:46,353 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:46,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718566368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718566369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,569 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/4b7840957ed140959b7379c5f356055b 2023-04-17 08:01:46,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718566673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718566673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,978 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/fd062526a23140548b25ea0ccff53e27 2023-04-17 08:01:46,981 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/3319610d0f654217b0933c7d941efabd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd 2023-04-17 08:01:46,984 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd, entries=150, sequenceid=275, filesize=25.4 K 2023-04-17 08:01:46,985 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/4b7840957ed140959b7379c5f356055b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b 2023-04-17 08:01:46,988 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b, entries=150, sequenceid=275, filesize=11.9 K 2023-04-17 08:01:46,988 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/fd062526a23140548b25ea0ccff53e27 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27 2023-04-17 08:01:46,991 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27, entries=150, sequenceid=275, filesize=11.9 K 2023-04-17 08:01:46,991 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c74f57daa4f72232bba5b12448b01e11 in 1284ms, sequenceid=275, compaction requested=false 2023-04-17 08:01:46,991 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:46,991 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:46,991 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:46,991 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:46,991 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:46,991 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:46,991 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:47,030 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:47,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,030 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:47,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:47,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3b5440[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:47,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:47,031 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,038 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:47,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:47,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:47,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:47,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:47,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:47,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:47,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:47,049 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:47,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:47,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:47,049 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:47,049 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:47,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,055 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:47,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:47,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:47,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:47,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:47,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:47,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:47,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:47,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:47,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:47,063 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:47,063 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:47,064 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:47,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:47,064 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:47,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:47,164 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:47,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:01:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:01:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:01:47,166 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:01:47,166 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:47,166 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:01:47,166 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:01:47,166 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,167 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:47,171 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:47,171 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:47,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:01:47,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:01:47,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:01:47,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:01:47,172 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:01:47,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:01:47,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:01:47,173 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. started... 2023-04-17 08:01:47,173 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:47,173 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:47,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:47,183 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. as already flushing 2023-04-17 08:01:47,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718567212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718567218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718567319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718567324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718567524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718567527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,542 DEBUG [Thread-2258] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4cc4 to 127.0.0.1:62477 2023-04-17 08:01:47,542 DEBUG [Thread-2258] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:47,543 DEBUG [Thread-2252] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52249391 to 127.0.0.1:62477 2023-04-17 08:01:47,543 DEBUG [Thread-2252] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:47,543 DEBUG [Thread-2256] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5144fcff to 127.0.0.1:62477 2023-04-17 08:01:47,544 DEBUG [Thread-2256] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:47,547 DEBUG [Thread-2254] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ba3ce53 to 127.0.0.1:62477 2023-04-17 08:01:47,548 DEBUG [Thread-2254] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:47,548 DEBUG [Thread-2260] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1280c068 to 127.0.0.1:62477 2023-04-17 08:01:47,548 DEBUG [Thread-2260] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:47,588 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c9c3032bfff44c0f95d3baae57e62586 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9c3032bfff44c0f95d3baae57e62586 2023-04-17 08:01:47,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718567830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718567834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37354 deadline: 1681718567929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,932 DEBUG [Thread-2245] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:47,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:37374 deadline: 1681718567942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:47,945 DEBUG [Thread-2249] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:01:47,995 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/2fed7442bac44d9a8848b50eb5495650 2023-04-17 08:01:48,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:48,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37364 deadline: 1681718568333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:01:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:37388 deadline: 1681718568340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,415 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/58b8f25896ca419f8d319ed102a4715e 2023-04-17 08:01:48,833 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/eb6df0c296ad4c71ae0e8dbb5060b983 2023-04-17 08:01:48,844 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/2fed7442bac44d9a8848b50eb5495650 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650 2023-04-17 08:01:48,848 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650, entries=150, sequenceid=291, filesize=25.4 K 2023-04-17 08:01:48,849 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/58b8f25896ca419f8d319ed102a4715e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e 2023-04-17 08:01:48,853 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e, entries=150, sequenceid=291, filesize=11.9 K 2023-04-17 08:01:48,854 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/eb6df0c296ad4c71ae0e8dbb5060b983 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983 2023-04-17 08:01:48,859 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983, entries=150, sequenceid=291, filesize=11.9 K 2023-04-17 08:01:48,860 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c74f57daa4f72232bba5b12448b01e11 in 1687ms, sequenceid=291, compaction requested=true 2023-04-17 08:01:48,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:48,860 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:48,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:01:48,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:01:48,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:01:48,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:01:48,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:48,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:01:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:48,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:01:48,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:01:48,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3689f36c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:01:48,890 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:01:48,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:01:48,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:01:48,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:01:48,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:01:48,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:48,905 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:01:48,905 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:01:48,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:01:48,907 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:01:48,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:01:48,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:01:48,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:01:48,907 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:01:48,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:01:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:01:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,913 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:01:48,913 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:01:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:01:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:01:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:01:48,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:01:48,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:48,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:01:48,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:01:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:01:48,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:01:48,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:01:48,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:01:48,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:01:48,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:48,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:01:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:01:48,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:01:48,922 DEBUG [Thread-2251] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:01:48,922 DEBUG [Thread-2251] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:01:48,922 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:01:49,022 DEBUG [Thread-2251] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:01:49,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:01:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:49,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:01:49,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:49,342 DEBUG [Thread-2243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d63a8ce to 127.0.0.1:62477 2023-04-17 08:01:49,343 DEBUG [Thread-2243] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:49,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:49,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:49,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:49,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:49,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:49,345 DEBUG [Thread-2241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eaa86ba to 127.0.0.1:62477 2023-04-17 08:01:49,345 DEBUG [Thread-2241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:49,768 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417395248cbed8f4a8390f7fe807013bb12 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417395248cbed8f4a8390f7fe807013bb12 2023-04-17 08:01:50,174 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/d117cdb4747d45e7b12df6ad2d492980 2023-04-17 08:01:50,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/461cc36ff6bc44868cc5bc512ecbb586 2023-04-17 08:01:51,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/4d1b77c69d3c457290a507861d5d97d8 2023-04-17 08:01:51,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/d117cdb4747d45e7b12df6ad2d492980 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980 2023-04-17 08:01:51,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980, entries=150, sequenceid=311, filesize=25.4 K 2023-04-17 08:01:51,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/461cc36ff6bc44868cc5bc512ecbb586 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586 2023-04-17 08:01:51,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586, entries=150, sequenceid=311, filesize=11.9 K 2023-04-17 08:01:51,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/4d1b77c69d3c457290a507861d5d97d8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8 2023-04-17 08:01:51,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8, entries=150, sequenceid=311, filesize=11.9 K 2023-04-17 08:01:51,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=6.71 KB/6870 for c74f57daa4f72232bba5b12448b01e11 in 1700ms, sequenceid=311, compaction requested=true 2023-04-17 08:01:51,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:51,042 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:51,042 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:51,043 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:51,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:51,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:51,044 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:51,044 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:51,044 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/A is initiating minor compaction (all files) 2023-04-17 08:01:51,044 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/B is initiating minor compaction (all files) 2023-04-17 08:01:51,045 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/A in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:51,045 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/B in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:51,045 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5cdbccefedde427782a97e5a52426ee6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=102.2 K 2023-04-17 08:01:51,045 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/3692253d7bcd40ea9365fad4d8ddd31e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=48.1 K 2023-04-17 08:01:51,045 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5cdbccefedde427782a97e5a52426ee6, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502730 2023-04-17 08:01:51,045 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3692253d7bcd40ea9365fad4d8ddd31e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502730 2023-04-17 08:01:51,046 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 3319610d0f654217b0933c7d941efabd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681718503893 2023-04-17 08:01:51,046 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4b7840957ed140959b7379c5f356055b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681718503893 2023-04-17 08:01:51,046 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2fed7442bac44d9a8848b50eb5495650, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681718506056 2023-04-17 08:01:51,046 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 58b8f25896ca419f8d319ed102a4715e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681718506056 2023-04-17 08:01:51,046 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d117cdb4747d45e7b12df6ad2d492980, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681718507212 2023-04-17 08:01:51,047 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 461cc36ff6bc44868cc5bc512ecbb586, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681718507212 2023-04-17 08:01:51,057 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#B#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:01:51,059 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#A#compaction#362 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:01:51,473 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/ed5e8aee59c9472aaf243c15beb87505 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/ed5e8aee59c9472aaf243c15beb87505 2023-04-17 08:01:51,479 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/B of c74f57daa4f72232bba5b12448b01e11 into ed5e8aee59c9472aaf243c15beb87505(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:51,479 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:51,479 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/B, priority=12, startTime=1681718511042; duration=0sec 2023-04-17 08:01:51,479 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:01:51,480 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:01:51,481 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:01:51,481 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): c74f57daa4f72232bba5b12448b01e11/C is initiating minor compaction (all files) 2023-04-17 08:01:51,481 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c74f57daa4f72232bba5b12448b01e11/C in TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:51,481 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/16e9b8b393ef4aa0b354ea927a8dc3d6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp, totalSize=48.1 K 2023-04-17 08:01:51,482 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 16e9b8b393ef4aa0b354ea927a8dc3d6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718502730 2023-04-17 08:01:51,482 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fd062526a23140548b25ea0ccff53e27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681718503893 2023-04-17 08:01:51,483 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting eb6df0c296ad4c71ae0e8dbb5060b983, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681718506056 2023-04-17 08:01:51,483 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4d1b77c69d3c457290a507861d5d97d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681718507212 2023-04-17 08:01:51,493 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): c74f57daa4f72232bba5b12448b01e11#C#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:01:51,907 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/f5dba956ebdf4684a3488b3742a8f113 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/f5dba956ebdf4684a3488b3742a8f113 2023-04-17 08:01:51,914 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/C of c74f57daa4f72232bba5b12448b01e11 into f5dba956ebdf4684a3488b3742a8f113(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:01:51,914 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:51,914 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/C, priority=12, startTime=1681718511043; duration=0sec 2023-04-17 08:01:51,914 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:52,301 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/cf9aba5248a54a9289f463e3ac0d38c1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cf9aba5248a54a9289f463e3ac0d38c1 2023-04-17 08:01:52,307 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c74f57daa4f72232bba5b12448b01e11/A of c74f57daa4f72232bba5b12448b01e11 into cf9aba5248a54a9289f463e3ac0d38c1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:01:52,308 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:52,308 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11., storeName=c74f57daa4f72232bba5b12448b01e11/A, priority=12, startTime=1681718511042; duration=1sec 2023-04-17 08:01:52,308 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:01:54,284 DEBUG [Thread-2247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ea1a2c1 to 127.0.0.1:62477 2023-04-17 08:01:54,284 DEBUG [Thread-2247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:57,971 DEBUG [Thread-2249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25185572 to 127.0.0.1:62477 2023-04-17 08:01:57,971 DEBUG [Thread-2249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:57,993 DEBUG [Thread-2245] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d80e5a8 to 127.0.0.1:62477 2023-04-17 08:01:57,993 DEBUG [Thread-2245] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:57,993 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2495 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7478 rows 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2487 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7461 rows 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2502 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7506 rows 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2501 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7503 rows 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2519 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7556 rows 2023-04-17 08:01:57,994 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:01:57,994 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377ba302 to 127.0.0.1:62477 2023-04-17 08:01:57,994 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:01:57,995 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:01:57,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:01:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:01:58,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:01:58,000 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718518000"}]},"ts":"1681718518000"} 2023-04-17 08:01:58,000 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:01:58,021 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:01:58,022 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, UNASSIGN}] 2023-04-17 08:01:58,023 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, UNASSIGN 2023-04-17 08:01:58,023 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:01:58,023 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718518023"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718518023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718518023"}]},"ts":"1681718518023"} 2023-04-17 08:01:58,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:01:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:01:58,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing c74f57daa4f72232bba5b12448b01e11, disabling compactions & flushes 2023-04-17 08:01:58,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. after waiting 0 ms 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:58,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing c74f57daa4f72232bba5b12448b01e11 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=A 2023-04-17 08:01:58,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:58,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=B 2023-04-17 08:01:58,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:58,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c74f57daa4f72232bba5b12448b01e11, store=C 2023-04-17 08:01:58,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:01:58,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:01:58,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c9b7200e6e334e7a94b9b72d79f4e797 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9b7200e6e334e7a94b9b72d79f4e797 2023-04-17 08:01:58,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:01:59,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/50cbf2bd9a90461190e8636ba953f600 2023-04-17 08:01:59,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:01:59,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/f832b0f3b8ab4a6c913f8d08980eb92f 2023-04-17 08:01:59,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/e9b6ab5b5b3543c182b0f24873daed71 2023-04-17 08:01:59,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/A/50cbf2bd9a90461190e8636ba953f600 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/50cbf2bd9a90461190e8636ba953f600 2023-04-17 08:01:59,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/50cbf2bd9a90461190e8636ba953f600, entries=150, sequenceid=321, filesize=25.4 K 2023-04-17 08:01:59,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/B/f832b0f3b8ab4a6c913f8d08980eb92f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/f832b0f3b8ab4a6c913f8d08980eb92f 2023-04-17 08:01:59,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/f832b0f3b8ab4a6c913f8d08980eb92f, entries=150, sequenceid=321, filesize=11.9 K 2023-04-17 08:01:59,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/.tmp/C/e9b6ab5b5b3543c182b0f24873daed71 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e9b6ab5b5b3543c182b0f24873daed71 2023-04-17 08:01:59,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e9b6ab5b5b3543c182b0f24873daed71, entries=150, sequenceid=321, filesize=11.9 K 2023-04-17 08:01:59,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c74f57daa4f72232bba5b12448b01e11 in 1672ms, sequenceid=321, compaction requested=false 2023-04-17 08:01:59,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5cdbccefedde427782a97e5a52426ee6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980] to archive 2023-04-17 08:01:59,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:59,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5cdbccefedde427782a97e5a52426ee6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/5cdbccefedde427782a97e5a52426ee6 2023-04-17 08:01:59,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/3319610d0f654217b0933c7d941efabd 2023-04-17 08:01:59,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/2fed7442bac44d9a8848b50eb5495650 2023-04-17 08:01:59,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/d117cdb4747d45e7b12df6ad2d492980 2023-04-17 08:01:59,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/3692253d7bcd40ea9365fad4d8ddd31e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586] to archive 2023-04-17 08:01:59,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:59,861 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/3692253d7bcd40ea9365fad4d8ddd31e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/3692253d7bcd40ea9365fad4d8ddd31e 2023-04-17 08:01:59,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/4b7840957ed140959b7379c5f356055b 2023-04-17 08:01:59,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/58b8f25896ca419f8d319ed102a4715e 2023-04-17 08:01:59,865 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/461cc36ff6bc44868cc5bc512ecbb586 2023-04-17 08:01:59,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb1a0bb980574b3099a861fa98100338, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/16e9b8b393ef4aa0b354ea927a8dc3d6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8] to archive 2023-04-17 08:01:59,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:01:59,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb1a0bb980574b3099a861fa98100338 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb1a0bb980574b3099a861fa98100338 2023-04-17 08:01:59,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/7c3ee940bca14eceb8886ea8a50e2cf7 2023-04-17 08:01:59,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/2db06bf43e2a4269b52f5be26a84aaef 2023-04-17 08:01:59,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/16e9b8b393ef4aa0b354ea927a8dc3d6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/16e9b8b393ef4aa0b354ea927a8dc3d6 2023-04-17 08:01:59,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/1a8c2762a90147a28a07038be850fa2f 2023-04-17 08:01:59,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/fd062526a23140548b25ea0ccff53e27 2023-04-17 08:01:59,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/eb6df0c296ad4c71ae0e8dbb5060b983 2023-04-17 08:01:59,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/4d1b77c69d3c457290a507861d5d97d8 2023-04-17 08:01:59,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits/324.seqid, newMaxSeqId=324, maxSeqId=4 2023-04-17 08:01:59,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11. 2023-04-17 08:01:59,884 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for c74f57daa4f72232bba5b12448b01e11: 2023-04-17 08:01:59,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:01:59,885 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c74f57daa4f72232bba5b12448b01e11, regionState=CLOSED 2023-04-17 08:01:59,886 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718519885"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718519885"}]},"ts":"1681718519885"} 2023-04-17 08:01:59,887 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-17 08:01:59,887 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure c74f57daa4f72232bba5b12448b01e11, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.8620 sec 2023-04-17 08:01:59,888 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-17 08:01:59,888 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c74f57daa4f72232bba5b12448b01e11, UNASSIGN in 1.8650 sec 2023-04-17 08:01:59,889 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718519889"}]},"ts":"1681718519889"} 2023-04-17 08:01:59,889 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:01:59,915 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:01:59,917 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9200 sec 2023-04-17 08:02:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 08:02:00,107 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-17 08:02:00,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:02:00,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,112 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,114 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 08:02:00,121 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:02:00,124 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits] 2023-04-17 08:02:00,128 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/50cbf2bd9a90461190e8636ba953f600 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/50cbf2bd9a90461190e8636ba953f600 2023-04-17 08:02:00,129 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cf9aba5248a54a9289f463e3ac0d38c1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/A/cf9aba5248a54a9289f463e3ac0d38c1 2023-04-17 08:02:00,133 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/ed5e8aee59c9472aaf243c15beb87505 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/ed5e8aee59c9472aaf243c15beb87505 2023-04-17 08:02:00,135 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/f832b0f3b8ab4a6c913f8d08980eb92f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/B/f832b0f3b8ab4a6c913f8d08980eb92f 2023-04-17 08:02:00,139 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e9b6ab5b5b3543c182b0f24873daed71 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/e9b6ab5b5b3543c182b0f24873daed71 2023-04-17 08:02:00,141 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/f5dba956ebdf4684a3488b3742a8f113 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/C/f5dba956ebdf4684a3488b3742a8f113 2023-04-17 08:02:00,145 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits/324.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11/recovered.edits/324.seqid 2023-04-17 08:02:00,146 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/c74f57daa4f72232bba5b12448b01e11 2023-04-17 08:02:00,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:02:00,146 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:02:00,147 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 08:02:00,152 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417053dbda3f3e14bec9f6672d2f8e3c32b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417053dbda3f3e14bec9f6672d2f8e3c32b 2023-04-17 08:02:00,154 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417201b33d97ff24fed8a5f7d90615bab99 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417201b33d97ff24fed8a5f7d90615bab99 2023-04-17 08:02:00,156 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041723219b44a44442f681aa2d306d2e039a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041723219b44a44442f681aa2d306d2e039a 2023-04-17 08:02:00,158 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172905cff20bf94faa9e87b53c7e7a15d0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172905cff20bf94faa9e87b53c7e7a15d0 2023-04-17 08:02:00,160 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417395248cbed8f4a8390f7fe807013bb12 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417395248cbed8f4a8390f7fe807013bb12 2023-04-17 08:02:00,162 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e2df2fdc8b74d2681d4bb9f4ef9f9e4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e2df2fdc8b74d2681d4bb9f4ef9f9e4 2023-04-17 08:02:00,163 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175838cdd221e942cdb2f97eb6871635d5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175838cdd221e942cdb2f97eb6871635d5 2023-04-17 08:02:00,165 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176e9653d51bfa499485f36f803aaa9982 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176e9653d51bfa499485f36f803aaa9982 2023-04-17 08:02:00,167 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796394aa0bb1c43fdbca21236256a4d4a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796394aa0bb1c43fdbca21236256a4d4a 2023-04-17 08:02:00,169 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041799bc20718ce7420aa2e6ed43fac49e88 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041799bc20718ce7420aa2e6ed43fac49e88 2023-04-17 08:02:00,170 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179c3070dae134475cb46fc629f913474f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179c3070dae134475cb46fc629f913474f 2023-04-17 08:02:00,172 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a9940fe3057e4403aff3aa2c3585bec9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a9940fe3057e4403aff3aa2c3585bec9 2023-04-17 08:02:00,174 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b3f35f9595a34ccc913801644b97eb11 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b3f35f9595a34ccc913801644b97eb11 2023-04-17 08:02:00,176 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b46d5c78322241288c151c537b955bd9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b46d5c78322241288c151c537b955bd9 2023-04-17 08:02:00,179 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9b7200e6e334e7a94b9b72d79f4e797 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9b7200e6e334e7a94b9b72d79f4e797 2023-04-17 08:02:00,180 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9c3032bfff44c0f95d3baae57e62586 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9c3032bfff44c0f95d3baae57e62586 2023-04-17 08:02:00,182 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ce5301a88a84417b85b6785366da32e2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ce5301a88a84417b85b6785366da32e2 2023-04-17 08:02:00,183 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cf7ac500f3724ad6a56c75b6b81064ea to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cf7ac500f3724ad6a56c75b6b81064ea 2023-04-17 08:02:00,185 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8554fda3bd044d2a94f135d463c41e9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8554fda3bd044d2a94f135d463c41e9 2023-04-17 08:02:00,186 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:02:00,188 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,190 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:02:00,192 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:02:00,193 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,193 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:02:00,193 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718520193"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:00,195 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:02:00,195 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c74f57daa4f72232bba5b12448b01e11, NAME => 'TestAcidGuarantees,,1681718485317.c74f57daa4f72232bba5b12448b01e11.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:02:00,195 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:02:00,195 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718520195"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:00,197 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:02:00,205 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 97 msec 2023-04-17 08:02:00,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 08:02:00,216 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-17 08:02:00,232 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=503 (was 522), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=281 (was 319), ProcessCount=167 (was 168), AvailableMemoryMB=7638 (was 7522) - AvailableMemoryMB LEAK? - 2023-04-17 08:02:00,243 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=503, MaxFileDescriptor=60000, SystemLoadAverage=281, ProcessCount=167, AvailableMemoryMB=7638 2023-04-17 08:02:00,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:02:00,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:02:00,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:00,248 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 08:02:00,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-17 08:02:00,248 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 08:02:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:00,249 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:00,250 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d empty. 2023-04-17 08:02:00,250 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:00,250 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:02:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:00,657 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:02:00,659 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 614ac39c30038f7f3ad5b0f592d6fa7d, NAME => 'TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 08:02:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:01,080 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:02:01,080 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 614ac39c30038f7f3ad5b0f592d6fa7d, disabling compactions & flushes 2023-04-17 08:02:01,080 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,080 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,080 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. after waiting 0 ms 2023-04-17 08:02:01,080 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,081 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:01,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 08:02:01,089 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718521089"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718521089"}]},"ts":"1681718521089"} 2023-04-17 08:02:01,091 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 08:02:01,093 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 08:02:01,093 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718521093"}]},"ts":"1681718521093"} 2023-04-17 08:02:01,094 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 08:02:01,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, ASSIGN}] 2023-04-17 08:02:01,125 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, ASSIGN 2023-04-17 08:02:01,127 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 08:02:01,278 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=614ac39c30038f7f3ad5b0f592d6fa7d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:01,278 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718521278"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718521278"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718521278"}]},"ts":"1681718521278"} 2023-04-17 08:02:01,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:01,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:01,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 614ac39c30038f7f3ad5b0f592d6fa7d, NAME => 'TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:02:01,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:02:01,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,441 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,443 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A 2023-04-17 08:02:01,443 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A 2023-04-17 08:02:01,443 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:01,443 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 614ac39c30038f7f3ad5b0f592d6fa7d columnFamilyName A 2023-04-17 08:02:01,444 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(310): Store=614ac39c30038f7f3ad5b0f592d6fa7d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:01,444 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,446 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B 2023-04-17 08:02:01,446 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B 2023-04-17 08:02:01,446 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:01,446 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 614ac39c30038f7f3ad5b0f592d6fa7d columnFamilyName B 2023-04-17 08:02:01,447 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(310): Store=614ac39c30038f7f3ad5b0f592d6fa7d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:01,447 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,449 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C 2023-04-17 08:02:01,449 DEBUG [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C 2023-04-17 08:02:01,449 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:01,449 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 614ac39c30038f7f3ad5b0f592d6fa7d columnFamilyName C 2023-04-17 08:02:01,449 INFO [StoreOpener-614ac39c30038f7f3ad5b0f592d6fa7d-1] regionserver.HStore(310): Store=614ac39c30038f7f3ad5b0f592d6fa7d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:01,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:02:01,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:01,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 08:02:01,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 614ac39c30038f7f3ad5b0f592d6fa7d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59482967, jitterRate=-0.11363472044467926}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:02:01,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:01,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., pid=49, masterSystemTime=1681718521431 2023-04-17 08:02:01,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:01,457 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=614ac39c30038f7f3ad5b0f592d6fa7d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:01,457 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718521457"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718521457"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718521457"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718521457"}]},"ts":"1681718521457"} 2023-04-17 08:02:01,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-17 08:02:01,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 in 179 msec 2023-04-17 08:02:01,460 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-17 08:02:01,460 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, ASSIGN in 338 msec 2023-04-17 08:02:01,461 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 08:02:01,461 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718521461"}]},"ts":"1681718521461"} 2023-04-17 08:02:01,462 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 08:02:01,497 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 08:02:01,498 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2510 sec 2023-04-17 08:02:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 08:02:02,359 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-17 08:02:02,361 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x55409eb8 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,379 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ea7ac79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,381 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,384 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,385 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 08:02:02,389 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:33438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 08:02:02,391 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x009b9721 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,405 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ff7cd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,406 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5a7ac4a1 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,421 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846250, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,422 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x682c0b9e to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,430 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@647f95f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,431 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x50d6899d to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,446 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58ae18e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,447 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x6662e946 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,464 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29c1e76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,465 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x2c240146 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,481 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ea9d51a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,482 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3d5f04c3 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,497 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fd1c1f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,497 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x6523d347 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,505 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@209faa3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,505 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x28072326 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,513 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5345748a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,513 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x2d90797d to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:02,522 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9f61fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:02,528 DEBUG [hconnection-0x3850d5ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:02,533 DEBUG [hconnection-0x3e47483b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,533 DEBUG [hconnection-0x1a05e668-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,533 DEBUG [hconnection-0x6405d14b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,533 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,535 DEBUG [hconnection-0x6c5a9742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,535 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,535 DEBUG [hconnection-0x716362c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,536 DEBUG [hconnection-0x1346db06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,536 DEBUG [hconnection-0x524db28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,536 DEBUG [hconnection-0x662cd978-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,536 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,537 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,544 DEBUG [hconnection-0x568f245-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:02,544 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,544 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,544 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,547 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45960, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,544 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45946, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:02,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:02,550 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:02,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:02,551 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:02,552 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,553 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45970, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:02,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:02,560 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 08:02:02,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:02,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:02,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:02,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:02,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:02,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:02,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718582567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718582567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718582568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718582568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718582568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:02,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:02,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:02,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:02,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:02,579 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:02,579 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:02,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:02,580 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:02,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:02,580 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,583 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:02,583 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:02,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:02,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495c608d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:02,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:02,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:02,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:02,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:02,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:02,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:02,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:02,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:02,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:02,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:02,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:02,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:02,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,621 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:02,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,622 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:02,622 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,722 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:02,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:02,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:02,728 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:02,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:02,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:02,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,728 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,737 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,738 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,738 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:02,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:02,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:02,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:02,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:02,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:02,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:02,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:02,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:02,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b52991[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:02,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,754 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,754 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,754 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:02,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:02,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:02,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,762 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:02,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:02,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:02,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:02,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:02,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:02,766 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:02,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,772 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:02,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:02,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:02,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,779 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:02,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,779 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:02,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718582873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718582873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718582873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718582873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:02,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718582876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,880 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:02,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:02,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:02,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:02,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:02,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:02,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,884 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,916 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,917 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,917 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:02,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:02,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:02,917 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:02,917 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:02,917 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:02,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:02,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:02,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:02,918 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:02,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:02,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@501af853[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:02,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:02,931 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:02,938 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:02,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:02,938 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:02,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:02,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:02,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:02,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:02,949 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:02,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:02,949 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:02,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:02,956 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,957 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:02,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:02,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:02,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:02,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:02,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:02,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:02,963 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:02,962 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:02,963 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:02,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:02,963 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:02,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:02,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/a611a4584ad24236ba1bbe57d86b12b4 2023-04-17 08:02:03,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ad47735989f2431e8d4c3ec07571ce0e 2023-04-17 08:02:03,063 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:03,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:03,066 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:03,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:03,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:03,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,066 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:03,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:03,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:03,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:03,072 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:03,072 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:03,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:03,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@297163af[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:03,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:03,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:03,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:03,088 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:03,087 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:03,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:03,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:03,098 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:03,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:03,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:03,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:03,098 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:03,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:03,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:03,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:03,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,113 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:03,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,113 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:03,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,113 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718583177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718583177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718583178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718583178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718583179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,213 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:03,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:03,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:03,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:03,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:03,216 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:03,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:03,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:03,216 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,217 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:03,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:03,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:03,222 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,222 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:03,222 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:03,222 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:03,222 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:03,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:03,223 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:03,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:03,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:03,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3032604[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:03,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:03,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:03,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:03,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:03,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:03,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:03,246 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:03,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:03,248 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:03,248 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:03,248 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:03,248 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:03,248 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:03,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,255 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:03,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:03,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:03,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,263 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:03,263 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,263 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,363 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:03,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:03,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:03,368 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:03,368 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,368 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:03,368 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:03,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,369 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,378 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,378 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:03,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:03,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:03,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:03,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,379 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:03,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:03,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:03,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:03,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:03,408 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,409 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,409 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,409 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:03,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:03,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb56e54[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:03,410 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/8808ed3a6a504260be29ce718ae7a35b 2023-04-17 08:02:03,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/a611a4584ad24236ba1bbe57d86b12b4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4 2023-04-17 08:02:03,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:03,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:03,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:03,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:03,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4, entries=150, sequenceid=14, filesize=11.6 K 2023-04-17 08:02:03,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ad47735989f2431e8d4c3ec07571ce0e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e 2023-04-17 08:02:03,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e, entries=150, sequenceid=14, filesize=11.6 K 2023-04-17 08:02:03,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/8808ed3a6a504260be29ce718ae7a35b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b 2023-04-17 08:02:03,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b, entries=150, sequenceid=14, filesize=11.6 K 2023-04-17 08:02:03,428 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 614ac39c30038f7f3ad5b0f592d6fa7d in 868ms, sequenceid=14, compaction requested=false 2023-04-17 08:02:03,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:03,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:03,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:03,431 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:03,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:03,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:03,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:03,431 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:03,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:03,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:03,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:03,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,446 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:03,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:03,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:03,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:03,446 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:03,447 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:03,547 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:03,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:03,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:03,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:03,549 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:03,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:03,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:03,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,550 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,554 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:03,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:03,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:03,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:03,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:03,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:03,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:03,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:03,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:03,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:03,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:03,556 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:03,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:03,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:03,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:03,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718583688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718583689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718583689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718583693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718583695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718583796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718583797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718583797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718583797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718583799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:03,963 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e5fb6ce60de6412ba7d4838dd9f6f6b0 2023-04-17 08:02:03,972 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/e0c382cf693b42eb92ebfe8e356397bb 2023-04-17 08:02:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718584000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718584000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718584000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718584001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718584001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718584304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718584304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718584305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718584305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718584306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,381 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b7ad3e7ae03845c58e9b28c4bae3d9d7 2023-04-17 08:02:04,384 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e5fb6ce60de6412ba7d4838dd9f6f6b0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0 2023-04-17 08:02:04,386 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0, entries=150, sequenceid=34, filesize=11.6 K 2023-04-17 08:02:04,387 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/e0c382cf693b42eb92ebfe8e356397bb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb 2023-04-17 08:02:04,390 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb, entries=150, sequenceid=34, filesize=11.6 K 2023-04-17 08:02:04,390 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b7ad3e7ae03845c58e9b28c4bae3d9d7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7 2023-04-17 08:02:04,394 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7, entries=150, sequenceid=34, filesize=11.6 K 2023-04-17 08:02:04,395 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 614ac39c30038f7f3ad5b0f592d6fa7d in 838ms, sequenceid=34, compaction requested=false 2023-04-17 08:02:04,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:04,395 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:04,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:04,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:04,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:04,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:04,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:04,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:04,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:04,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6acff1[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:04,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:04,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:04,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:04,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:04,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:04,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:04,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:04,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:04,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:04,423 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:04,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:04,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:04,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:04,423 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:04,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:04,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:04,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:04,437 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:04,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:04,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:04,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:04,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:04,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:04,438 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:04,438 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:04,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:04,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:04,438 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:04,538 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:04,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:04,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:04,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:04,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:04,541 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:04,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:04,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:04,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:04,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,541 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:04,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:04,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:04,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:04,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:04,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:04,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:04,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:04,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:04,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:04,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:04,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:04,547 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:04,547 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:04,547 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:04,547 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:04,548 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:04,553 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/3b9aca90a839463c84a7cc182957a374 2023-04-17 08:02:04,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:04,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:04,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718584832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718584836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718584837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718584837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718584838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718584940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718584940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718584946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718584946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718584946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:04,962 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/cbf811d754184c8e83babd4eb432ed88 2023-04-17 08:02:05,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718585144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718585145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718585149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718585149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718585150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,370 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/dfc1d0fb827e4fdfaa759341b26cf6b3 2023-04-17 08:02:05,373 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/3b9aca90a839463c84a7cc182957a374 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374 2023-04-17 08:02:05,376 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374, entries=100, sequenceid=46, filesize=9.3 K 2023-04-17 08:02:05,376 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/cbf811d754184c8e83babd4eb432ed88 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88 2023-04-17 08:02:05,379 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88, entries=100, sequenceid=46, filesize=9.3 K 2023-04-17 08:02:05,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/dfc1d0fb827e4fdfaa759341b26cf6b3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3 2023-04-17 08:02:05,383 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3, entries=100, sequenceid=46, filesize=9.3 K 2023-04-17 08:02:05,383 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 614ac39c30038f7f3ad5b0f592d6fa7d in 836ms, sequenceid=46, compaction requested=true 2023-04-17 08:02:05,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:05,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:05,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:05,384 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,384 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:05,384 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:05,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:05,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:05,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6924a128[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:05,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:05,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:05,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:05,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:05,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:05,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:05,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:05,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:05,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:05,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:05,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:05,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,413 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 08:02:05,414 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,415 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:05,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,429 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:05,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,429 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:05,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:05,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:05,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:05,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/0747f42b53144d1eabfa1a483683391f 2023-04-17 08:02:05,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718585461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718585461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718585462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718585462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718585463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,530 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:05,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:05,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:05,532 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:05,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:05,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:05,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:05,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:05,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:05,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:05,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:05,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:05,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:05,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:05,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:05,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:05,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:05,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:05,540 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:05,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:05,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738d890e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:05,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:05,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,554 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:05,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:05,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:05,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:05,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718585570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718585570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718585570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718585572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718585572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,576 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:05,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,576 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,576 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:05,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:05,579 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:05,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:05,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:05,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:05,579 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:05,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:05,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:05,596 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,596 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,696 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:05,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:05,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:05,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:05,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:05,699 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:05,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:05,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:05,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,700 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:05,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:05,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:05,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:05,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:05,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:05,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:05,706 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:05,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:05,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142d8ee5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:05,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:05,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,723 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:05,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:05,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:05,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:05,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:05,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:05,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:05,729 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:05,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:05,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:05,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:05,730 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:05,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:05,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,746 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,746 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:05,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718585776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718585776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718585776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718585776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718585778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,846 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:05,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:05,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:05,849 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:05,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:05,849 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:05,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,854 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,854 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:05,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:05,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:05,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:05,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:05,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:05,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:05,855 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:05,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:05,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:05,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:05,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:05,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:05,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:05,862 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a5ded6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:05,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:05,864 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/3b39cb8ece53438ca5c04c07ecb6d341 2023-04-17 08:02:05,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:05,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:05,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:05,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:05,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,879 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:05,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,880 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:05,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:05,881 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:05,881 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:05,881 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:05,881 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:05,881 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:05,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:05,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:05,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:05,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:05,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:05,896 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:05,896 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:05,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:05,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,996 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:05,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:05,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:05,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:05,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:05,999 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:05,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:05,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:05,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:05,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:05,999 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:06,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:06,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:06,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:06,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:06,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:06,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:06,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:06,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4544032b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:06,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:06,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:06,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:06,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:06,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:06,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:06,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:06,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:06,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:06,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:06,061 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,061 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,061 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,061 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:06,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,071 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:06,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:06,071 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718586081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718586081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718586081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718586082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718586082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,171 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:06,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:06,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:06,173 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:06,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:06,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:06,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:06,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:06,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:06,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:06,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:06,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:06,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:06,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:06,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:06,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:06,181 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:06,181 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:06,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:06,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9e3774[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:06,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:06,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,196 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:06,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:06,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:06,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:06,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:06,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:06,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:06,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:06,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:06,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:06,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:06,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:06,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:06,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:06,221 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:06,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,221 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/cc22b86b172241e795ade69d0044525f 2023-04-17 08:02:06,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/0747f42b53144d1eabfa1a483683391f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f 2023-04-17 08:02:06,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f, entries=200, sequenceid=67, filesize=13.9 K 2023-04-17 08:02:06,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/3b39cb8ece53438ca5c04c07ecb6d341 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341 2023-04-17 08:02:06,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341, entries=150, sequenceid=67, filesize=11.6 K 2023-04-17 08:02:06,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/cc22b86b172241e795ade69d0044525f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f 2023-04-17 08:02:06,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f, entries=150, sequenceid=67, filesize=11.6 K 2023-04-17 08:02:06,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 614ac39c30038f7f3ad5b0f592d6fa7d in 839ms, sequenceid=67, compaction requested=true 2023-04-17 08:02:06,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:06,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:06,293 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:06,293 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:06,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:06,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:06,294 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:06,295 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,295 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=46.4 K 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a611a4584ad24236ba1bbe57d86b12b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681718522555 2023-04-17 08:02:06,295 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,295 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=44.1 K 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e5fb6ce60de6412ba7d4838dd9f6f6b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681718522566 2023-04-17 08:02:06,295 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ad47735989f2431e8d4c3ec07571ce0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681718522555 2023-04-17 08:02:06,296 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 3b9aca90a839463c84a7cc182957a374, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681718523692 2023-04-17 08:02:06,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e0c382cf693b42eb92ebfe8e356397bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681718522566 2023-04-17 08:02:06,296 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 0747f42b53144d1eabfa1a483683391f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:06,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting cbf811d754184c8e83babd4eb432ed88, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681718523692 2023-04-17 08:02:06,296 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3b39cb8ece53438ca5c04c07ecb6d341, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:06,311 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#380 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:06,321 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:06,323 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:06,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:06,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:06,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:06,341 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:06,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:06,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:06,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:06,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:06,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:06,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:06,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:06,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:06,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:06,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:06,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:06,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:06,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:06,347 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 08:02:06,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:06,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:06,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:06,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:06,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:06,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:06,386 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/7436f5d3b97541e3a9793f4cd85e2739 2023-04-17 08:02:06,428 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/14565b20c8d24528995e469ea9173652 2023-04-17 08:02:06,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:06,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:06,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718586610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718586611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718526605, max timestamp=1681718526616Num uniques 859; 2023-04-17 08:02:06,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:02:06,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718526605, max timestamp=1681718526616Num uniques 859; 2023-04-17 08:02:06,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718526605, max timestamp=1681718526616Num uniques 859; 2023-04-17 08:02:06,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:02:06,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:02:06,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718586640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718586642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718586646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718586718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718586719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,745 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/810cdad1e8cf40ebbac25a3fd94108e7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/810cdad1e8cf40ebbac25a3fd94108e7 2023-04-17 08:02:06,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718586746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,751 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into 810cdad1e8cf40ebbac25a3fd94108e7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:06,751 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:06,751 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=12, startTime=1681718526293; duration=0sec 2023-04-17 08:02:06,751 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:06,751 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:06,753 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,753 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=44.1 K 2023-04-17 08:02:06,753 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 8808ed3a6a504260be29ce718ae7a35b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681718522555 2023-04-17 08:02:06,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718586748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,754 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b7ad3e7ae03845c58e9b28c4bae3d9d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681718522566 2023-04-17 08:02:06,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718586749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,755 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting dfc1d0fb827e4fdfaa759341b26cf6b3, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681718523692 2023-04-17 08:02:06,755 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting cc22b86b172241e795ade69d0044525f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:06,756 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/50bf93ba7dd848da8930c80129367b64 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/50bf93ba7dd848da8930c80129367b64 2023-04-17 08:02:06,759 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 50bf93ba7dd848da8930c80129367b64(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:06,759 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:06,759 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=12, startTime=1681718526293; duration=0sec 2023-04-17 08:02:06,759 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:06,762 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#385 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:06,870 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/aff9d448c99645a9b64aa813b033b012 2023-04-17 08:02:06,875 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/7436f5d3b97541e3a9793f4cd85e2739 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739 2023-04-17 08:02:06,880 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739, entries=100, sequenceid=78, filesize=9.3 K 2023-04-17 08:02:06,881 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/14565b20c8d24528995e469ea9173652 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652 2023-04-17 08:02:06,884 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652, entries=100, sequenceid=78, filesize=9.3 K 2023-04-17 08:02:06,884 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/aff9d448c99645a9b64aa813b033b012 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012 2023-04-17 08:02:06,889 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012, entries=100, sequenceid=78, filesize=9.3 K 2023-04-17 08:02:06,889 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 614ac39c30038f7f3ad5b0f592d6fa7d in 542ms, sequenceid=78, compaction requested=false 2023-04-17 08:02:06,890 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:06,890 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:06,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:06,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:06,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:06,890 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:06,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:06,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:06,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22bcc064[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:06,907 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:06,913 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:06,912 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,913 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:06,913 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:06,921 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:06,921 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,921 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:06,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:06,923 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:06,923 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:06,923 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:06,923 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:06,923 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:06,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:06,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:06,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-17 08:02:06,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:06,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:06,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:06,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,932 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:06,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:06,945 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:06,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:06,945 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:06,945 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:06,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:06,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:06,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718586947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718586953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718586953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718586955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:06,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:06,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718586955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,045 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:07,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:07,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:07,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:07,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:07,048 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:07,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:07,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:07,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,048 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:07,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:07,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:07,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:07,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:07,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:07,055 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:07,055 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,055 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:07,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:07,056 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:07,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:07,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:07,056 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:07,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718587054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:07,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@548f98b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:07,064 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:07,064 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718587061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:07,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:07,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:07,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:07,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:07,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:07,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:07,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:07,081 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:07,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:07,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:07,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:07,081 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:07,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:07,138 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,138 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,173 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/e2f2d6b8f35348eabe8c277dd1182b71 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/e2f2d6b8f35348eabe8c277dd1182b71 2023-04-17 08:02:07,176 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into e2f2d6b8f35348eabe8c277dd1182b71(size=11.7 K), total size for store is 21.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:07,176 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,176 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=12, startTime=1681718526294; duration=0sec 2023-04-17 08:02:07,176 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:07,238 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:07,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:07,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:07,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:07,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:07,241 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:07,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:07,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:07,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:07,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:07,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:07,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:07,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:07,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:07,255 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,255 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:07,255 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:07,256 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:07,256 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:07,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:07,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:07,256 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:07,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718587259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718587260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718587261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:07,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6070ef52[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:07,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:07,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718587262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718587266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:07,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:07,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:07,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:07,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:07,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:07,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:07,281 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:07,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:07,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:07,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:07,281 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:07,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:07,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:07,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,296 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,296 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/fd466b9a98ae4e7e81a3c8de3951889d 2023-04-17 08:02:07,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/d2c9144530214075a56f15e156d65c27 2023-04-17 08:02:07,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/49e05a9b293a4283b95d148c7e74146c 2023-04-17 08:02:07,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/fd466b9a98ae4e7e81a3c8de3951889d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d 2023-04-17 08:02:07,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d, entries=200, sequenceid=115, filesize=13.9 K 2023-04-17 08:02:07,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/d2c9144530214075a56f15e156d65c27 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27 2023-04-17 08:02:07,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 08:02:07,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/49e05a9b293a4283b95d148c7e74146c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c 2023-04-17 08:02:07,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 08:02:07,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 614ac39c30038f7f3ad5b0f592d6fa7d in 464ms, sequenceid=115, compaction requested=true 2023-04-17 08:02:07,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:07,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:07,395 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:07,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:07,395 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:07,396 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,396 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,396 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/810cdad1e8cf40ebbac25a3fd94108e7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=32.7 K 2023-04-17 08:02:07,396 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/50bf93ba7dd848da8930c80129367b64, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=35.0 K 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 810cdad1e8cf40ebbac25a3fd94108e7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 50bf93ba7dd848da8930c80129367b64, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:07,396 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:07,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 14565b20c8d24528995e469ea9173652, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681718525461 2023-04-17 08:02:07,397 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7436f5d3b97541e3a9793f4cd85e2739, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681718525461 2023-04-17 08:02:07,397 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d2c9144530214075a56f15e156d65c27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:07,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:07,397 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting fd466b9a98ae4e7e81a3c8de3951889d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:07,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:07,400 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:07,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:07,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:07,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,400 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,402 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#389 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:07,404 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:07,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:07,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:07,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:07,425 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:07,425 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:07,425 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:07,425 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:07,425 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:07,425 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:07,426 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:07,426 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:07,432 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/453a21c4a461402ca81c8b89e0ec4089 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/453a21c4a461402ca81c8b89e0ec4089 2023-04-17 08:02:07,436 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/c41600e9bbf146dc92ee025223b29e90 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/c41600e9bbf146dc92ee025223b29e90 2023-04-17 08:02:07,437 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into 453a21c4a461402ca81c8b89e0ec4089(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:07,438 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,438 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=13, startTime=1681718527395; duration=0sec 2023-04-17 08:02:07,438 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:07,438 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:07,438 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:07,438 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:07,439 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,439 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/e2f2d6b8f35348eabe8c277dd1182b71, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=32.7 K 2023-04-17 08:02:07,439 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e2f2d6b8f35348eabe8c277dd1182b71, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681718524830 2023-04-17 08:02:07,439 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting aff9d448c99645a9b64aa813b033b012, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681718525461 2023-04-17 08:02:07,439 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 49e05a9b293a4283b95d148c7e74146c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:07,440 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into c41600e9bbf146dc92ee025223b29e90(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:07,440 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,440 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=13, startTime=1681718527395; duration=0sec 2023-04-17 08:02:07,441 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:07,446 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/0678a21deb73400a890858dfa6245e9a 2023-04-17 08:02:07,465 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:07,488 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/cee525fae05847079bc82d9336f40604 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cee525fae05847079bc82d9336f40604 2023-04-17 08:02:07,494 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into cee525fae05847079bc82d9336f40604(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:07,494 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,494 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=13, startTime=1681718527395; duration=0sec 2023-04-17 08:02:07,494 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:07,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:07,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:07,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718527605, max timestamp=1681718527614Num uniques 859; 2023-04-17 08:02:07,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718527605, max timestamp=1681718527614Num uniques 859; 2023-04-17 08:02:07,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:07,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718527605, max timestamp=1681718527614Num uniques 859; 2023-04-17 08:02:07,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:07,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:07,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718587674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718587675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718587763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718587765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718587771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45954 deadline: 1681718587779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45926 deadline: 1681718587779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,897 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/6a0a2c53615b425480f4a54f29cf3226 2023-04-17 08:02:07,914 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/26c63ef398664a508a446abb1906c94b 2023-04-17 08:02:07,918 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/0678a21deb73400a890858dfa6245e9a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a 2023-04-17 08:02:07,925 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a, entries=150, sequenceid=126, filesize=11.6 K 2023-04-17 08:02:07,926 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/6a0a2c53615b425480f4a54f29cf3226 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226 2023-04-17 08:02:07,929 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226, entries=150, sequenceid=126, filesize=11.6 K 2023-04-17 08:02:07,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/26c63ef398664a508a446abb1906c94b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b 2023-04-17 08:02:07,940 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b, entries=150, sequenceid=126, filesize=11.6 K 2023-04-17 08:02:07,941 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 614ac39c30038f7f3ad5b0f592d6fa7d in 515ms, sequenceid=126, compaction requested=false 2023-04-17 08:02:07,941 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:07,941 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:07,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:07,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:07,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:07,942 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:07,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:07,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bfe41bf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:07,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:07,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,954 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:07,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:07,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:07,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:07,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:07,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,963 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:07,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:07,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:07,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:07,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:07,965 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:07,965 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:07,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:07,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:07,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718527635, max timestamp=1681718527989Num uniques 1018; 2023-04-17 08:02:07,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718527635, max timestamp=1681718527989Num uniques 1018; 2023-04-17 08:02:07,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 08:02:07,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718527635, max timestamp=1681718527989Num uniques 1018; 2023-04-17 08:02:07,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 08:02:07,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-17 08:02:07,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:07,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:07,995 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:07,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:07,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:07,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:07,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:07,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:07,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:07,995 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:07,996 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:07,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:07,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:07,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:07,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:07,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:07,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/604d58ee88b9447ab0e4cd152e5d444c 2023-04-17 08:02:08,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718528018, max timestamp=1681718528035Num uniques 859; 2023-04-17 08:02:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718528018, max timestamp=1681718528035Num uniques 859; 2023-04-17 08:02:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718528018, max timestamp=1681718528035Num uniques 859; 2023-04-17 08:02:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:08,096 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,098 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718528060, max timestamp=1681718528100Num uniques 1068; 2023-04-17 08:02:08,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:02:08,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718528060, max timestamp=1681718528100Num uniques 1068; 2023-04-17 08:02:08,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:02:08,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681718528060, max timestamp=1681718528100Num uniques 1068; 2023-04-17 08:02:08,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 08:02:08,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,104 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:08,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,106 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,106 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,113 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f851744[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,121 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,121 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,131 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,131 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,131 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,131 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,131 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718528124, max timestamp=1681718528145Num uniques 1027; 2023-04-17 08:02:08,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:02:08,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718528124, max timestamp=1681718528145Num uniques 1027; 2023-04-17 08:02:08,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:02:08,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681718528124, max timestamp=1681718528145Num uniques 1027; 2023-04-17 08:02:08,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 08:02:08,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,165 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,164 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,165 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,165 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718528181, max timestamp=1681718528199Num uniques 1136; 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718528181, max timestamp=1681718528199Num uniques 1136; 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718528181, max timestamp=1681718528199Num uniques 1136; 2023-04-17 08:02:08,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-17 08:02:08,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718528237, max timestamp=1681718528254Num uniques 1045; 2023-04-17 08:02:08,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718528237, max timestamp=1681718528254Num uniques 1045; 2023-04-17 08:02:08,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:02:08,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:02:08,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681718528237, max timestamp=1681718528254Num uniques 1045; 2023-04-17 08:02:08,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-17 08:02:08,265 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,267 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,267 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,268 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,268 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:08,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,281 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,281 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,281 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,281 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,281 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,281 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,287 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc3239e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,289 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681718528274, max timestamp=1681718528301Num uniques 1104; 2023-04-17 08:02:08,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-17 08:02:08,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681718528274, max timestamp=1681718528301Num uniques 1104; 2023-04-17 08:02:08,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-17 08:02:08,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681718528274, max timestamp=1681718528301Num uniques 1104; 2023-04-17 08:02:08,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-17 08:02:08,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,306 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,306 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,306 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,306 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,306 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,313 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,321 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,321 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,321 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,321 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718528344, max timestamp=1681718528358Num uniques 1113; 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718528344, max timestamp=1681718528358Num uniques 1113; 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681718528344, max timestamp=1681718528358Num uniques 1113; 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-17 08:02:08,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-17 08:02:08,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333 2023-04-17 08:02:08,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718528401, max timestamp=1681718528411Num uniques 1072; 2023-04-17 08:02:08,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-17 08:02:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718528401, max timestamp=1681718528411Num uniques 1072; 2023-04-17 08:02:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681718528401, max timestamp=1681718528411Num uniques 1072; 2023-04-17 08:02:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-17 08:02:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-17 08:02:08,421 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,424 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,424 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:08,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,430 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,430 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,430 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,430 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,430 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,431 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,431 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,437 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aae5f0b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,446 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,446 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,446 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,456 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,456 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,468 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,468 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,468 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,468 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,469 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718528451, max timestamp=1681718528474Num uniques 1081; 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718528451, max timestamp=1681718528474Num uniques 1081; 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681718528451, max timestamp=1681718528474Num uniques 1081; 2023-04-17 08:02:08,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,480 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,480 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681718528507, max timestamp=1681718528541Num uniques 1040; 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681718528507, max timestamp=1681718528541Num uniques 1040; 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681718528507, max timestamp=1681718528541Num uniques 1040; 2023-04-17 08:02:08,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-17 08:02:08,580 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,582 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,583 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:08,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,589 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,589 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,589 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,589 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,589 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e41058[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718528573, max timestamp=1681718528602Num uniques 1099; 2023-04-17 08:02:08,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-17 08:02:08,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718528573, max timestamp=1681718528602Num uniques 1099; 2023-04-17 08:02:08,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-17 08:02:08,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681718528573, max timestamp=1681718528602Num uniques 1099; 2023-04-17 08:02:08,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-17 08:02:08,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,612 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,612 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,612 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,614 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,614 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,629 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,629 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718528638, max timestamp=1681718528651Num uniques 1108; 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718528638, max timestamp=1681718528651Num uniques 1108; 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681718528638, max timestamp=1681718528651Num uniques 1108; 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-17 08:02:08,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-17 08:02:08,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681718528674, max timestamp=1681718528689Num uniques 1117; 2023-04-17 08:02:08,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681718528674, max timestamp=1681718528689Num uniques 1117; 2023-04-17 08:02:08,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-17 08:02:08,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-17 08:02:08,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681718528674, max timestamp=1681718528689Num uniques 1117; 2023-04-17 08:02:08,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-17 08:02:08,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681718528711, max timestamp=1681718528723Num uniques 1076; 2023-04-17 08:02:08,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-17 08:02:08,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681718528711, max timestamp=1681718528723Num uniques 1076; 2023-04-17 08:02:08,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-17 08:02:08,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681718528711, max timestamp=1681718528723Num uniques 1076; 2023-04-17 08:02:08,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-17 08:02:08,729 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,732 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,732 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,733 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,737 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,738 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,738 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:08,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,739 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,739 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681718528747, max timestamp=1681718528762Num uniques 1085; 2023-04-17 08:02:08,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-17 08:02:08,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681718528747, max timestamp=1681718528762Num uniques 1085; 2023-04-17 08:02:08,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-17 08:02:08,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681718528747, max timestamp=1681718528762Num uniques 1085; 2023-04-17 08:02:08,765 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,765 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@347c0a57[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,766 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681718528779, max timestamp=1681718528787Num uniques 1144; 2023-04-17 08:02:08,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-17 08:02:08,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681718528779, max timestamp=1681718528787Num uniques 1144; 2023-04-17 08:02:08,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-17 08:02:08,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681718528779, max timestamp=1681718528787Num uniques 1144; 2023-04-17 08:02:08,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-17 08:02:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,791 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,791 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,804 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,804 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,805 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,804 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718588813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718588813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718588815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681718528803, max timestamp=1681718528818Num uniques 1153; 2023-04-17 08:02:08,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681718528803, max timestamp=1681718528818Num uniques 1153; 2023-04-17 08:02:08,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-17 08:02:08,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681718528803, max timestamp=1681718528818Num uniques 1153; 2023-04-17 08:02:08,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-17 08:02:08,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-17 08:02:08,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/8500043867e04a7cb290eaab39ececac 2023-04-17 08:02:08,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/604d58ee88b9447ab0e4cd152e5d444c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c 2023-04-17 08:02:08,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c, entries=200, sequenceid=167, filesize=14.1 K 2023-04-17 08:02:08,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333 2023-04-17 08:02:08,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333, entries=150, sequenceid=167, filesize=11.8 K 2023-04-17 08:02:08,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/8500043867e04a7cb290eaab39ececac as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac 2023-04-17 08:02:08,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac, entries=150, sequenceid=167, filesize=11.8 K 2023-04-17 08:02:08,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=60.38 KB/61830 for 614ac39c30038f7f3ad5b0f592d6fa7d in 841ms, sequenceid=167, compaction requested=true 2023-04-17 08:02:08,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:08,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:08,836 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:08,836 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:08,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:08,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:08,836 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:08,837 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,837 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,837 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/c41600e9bbf146dc92ee025223b29e90, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=37.5 K 2023-04-17 08:02:08,837 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/453a21c4a461402ca81c8b89e0ec4089, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=35.2 K 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c41600e9bbf146dc92ee025223b29e90, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 453a21c4a461402ca81c8b89e0ec4089, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:08,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=73.80 KB heapSize=196.88 KB 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 6a0a2c53615b425480f4a54f29cf3226, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681718526944 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:08,837 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0678a21deb73400a890858dfa6245e9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681718526944 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:08,838 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1b08eaa1c74b4ddcb20d9e6ebe8c9333, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527672 2023-04-17 08:02:08,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:08,838 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 604d58ee88b9447ab0e4cd152e5d444c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527635 2023-04-17 08:02:08,843 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:08,843 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:08,854 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/68bd7086f81f4b398e84645573bb1377 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/68bd7086f81f4b398e84645573bb1377 2023-04-17 08:02:08,856 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/2ad46eac6b6b4cd6829d118bf56cc4f0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/2ad46eac6b6b4cd6829d118bf56cc4f0 2023-04-17 08:02:08,858 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into 68bd7086f81f4b398e84645573bb1377(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:08,858 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:08,858 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=13, startTime=1681718528836; duration=0sec 2023-04-17 08:02:08,858 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:08,858 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:08,859 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:08,859 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:08,859 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,859 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cee525fae05847079bc82d9336f40604, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=35.2 K 2023-04-17 08:02:08,860 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting cee525fae05847079bc82d9336f40604, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681718526631 2023-04-17 08:02:08,860 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 26c63ef398664a508a446abb1906c94b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681718526944 2023-04-17 08:02:08,860 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 8500043867e04a7cb290eaab39ececac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527672 2023-04-17 08:02:08,861 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 2ad46eac6b6b4cd6829d118bf56cc4f0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:08,861 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:08,861 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=13, startTime=1681718528836; duration=0sec 2023-04-17 08:02:08,861 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:08,865 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718588887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718588888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,905 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:08,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:08,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:08,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:08,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:08,916 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,916 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:08,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:08,922 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,922 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:08,922 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:08,922 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:08,922 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:08,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718588919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718588921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718588921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,929 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:08,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f70095[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:08,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:08,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:08,937 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:08,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,938 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:08,938 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:08,946 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,946 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:08,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:08,947 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:08,947 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:08,947 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:08,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:08,948 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:08,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:08,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,954 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:08,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:08,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:08,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:08,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:08,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:08,962 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:08,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:08,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:08,963 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:08,963 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:08,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718588995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:08,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:08,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718588995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,063 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,067 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,067 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,068 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,068 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,078 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,078 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,078 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,084 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,084 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,084 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,084 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,088 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,088 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38f95d87[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,096 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,106 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,106 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,106 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,113 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,114 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,121 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:09,121 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,121 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,121 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718589124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718589124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718589127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718589199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718589199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,221 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,224 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,224 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,230 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,230 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,230 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,231 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,231 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,237 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b2cec10[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e7b8f2adb23f44988f1825e5b2c289a9 2023-04-17 08:02:09,265 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,266 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,266 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,273 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/271785deaa234a239119eb823a8c5b93 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/271785deaa234a239119eb823a8c5b93 2023-04-17 08:02:09,276 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into 271785deaa234a239119eb823a8c5b93(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:09,276 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:09,276 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=13, startTime=1681718528836; duration=0sec 2023-04-17 08:02:09,276 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:09,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,280 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,280 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,297 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,298 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,304 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,305 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,305 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:09,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,306 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,306 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,406 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,408 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,408 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,409 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,412 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,414 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,415 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,415 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,415 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,415 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,415 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,422 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd66190[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718589428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718589429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718589431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,438 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,439 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,439 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,439 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,447 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,448 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,454 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,454 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:09,454 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,455 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718589504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718589504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,555 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,557 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,557 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,558 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,564 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,564 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,564 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,564 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,572 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd3157[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,574 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,574 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,589 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,589 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,590 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,604 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,604 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,605 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/fcc4874308b746549c9b8a733fddd73a 2023-04-17 08:02:09,705 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,707 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,707 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,708 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,708 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,712 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,713 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,714 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,714 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,714 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,714 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,714 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,721 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,721 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,721 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751a514a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,722 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,739 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,739 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,766 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,766 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,766 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,767 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,768 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:09,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,779 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,780 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,880 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:09,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:09,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:09,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:09,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:09,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:09,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:09,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:09,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:09,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:09,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:09,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:09,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:09,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:09,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:09,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:09,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:09,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:09,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:09,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:09,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ba7f95[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:09,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:09,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:09,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:09,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:09,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:09,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:09,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:09,915 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:09,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:09,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:09,915 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:09,915 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:09,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:09,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:09,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:09,929 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:09,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:09,929 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:09,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:09,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:09,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718589936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718589936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:09,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:09,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718589937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718590009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:10,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718590012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,030 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:10,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:10,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:10,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:10,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:10,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:10,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:10,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:10,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:10,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:10,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:10,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:10,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:10,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:10,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:10,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:10,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:10,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:10,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:10,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:10,039 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:10,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:10,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:10,052 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758c57d3[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:10,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:10,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:10,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:10,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:10,063 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:10,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ba52643fb8a0443fbe43249ef07ddda2 2023-04-17 08:02:10,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e7b8f2adb23f44988f1825e5b2c289a9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9 2023-04-17 08:02:10,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:10,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:10,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:10,072 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9, entries=150, sequenceid=468, filesize=11.9 K 2023-04-17 08:02:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:10,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/fcc4874308b746549c9b8a733fddd73a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a 2023-04-17 08:02:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:10,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:10,075 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:10,075 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:10,075 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:10,075 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:10,075 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:10,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a, entries=150, sequenceid=468, filesize=11.9 K 2023-04-17 08:02:10,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ba52643fb8a0443fbe43249ef07ddda2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2 2023-04-17 08:02:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:10,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:10,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:10,081 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2, entries=150, sequenceid=468, filesize=11.9 K 2023-04-17 08:02:10,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:10,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:10,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~196.83 KB/201552, currentSize=107.34 KB/109920 for 614ac39c30038f7f3ad5b0f592d6fa7d in 1250ms, sequenceid=468, compaction requested=false 2023-04-17 08:02:10,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:10,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:10,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:10,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:10,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:10,088 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:10,088 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:10,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,188 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:10,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:10,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:10,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:10,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:10,191 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:10,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:10,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:10,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:10,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,191 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,196 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:10,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:10,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:10,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:10,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:10,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:10,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:10,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:10,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:10,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:10,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:10,197 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:10,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:10,603 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/fe39986617594477b200a517d0b5d039 2023-04-17 08:02:10,611 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/1ab3b6f177cd4b61aca87ebbd8e09140 2023-04-17 08:02:10,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:10,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:10,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718590957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718590957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718590958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718591015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718591016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,022 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f047e0b274534d5193f953429f33d818 2023-04-17 08:02:11,024 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/fe39986617594477b200a517d0b5d039 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039 2023-04-17 08:02:11,027 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039, entries=150, sequenceid=490, filesize=11.9 K 2023-04-17 08:02:11,029 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/1ab3b6f177cd4b61aca87ebbd8e09140 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140 2023-04-17 08:02:11,032 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140, entries=150, sequenceid=490, filesize=11.9 K 2023-04-17 08:02:11,032 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f047e0b274534d5193f953429f33d818 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818 2023-04-17 08:02:11,035 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818, entries=150, sequenceid=490, filesize=11.9 K 2023-04-17 08:02:11,035 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 614ac39c30038f7f3ad5b0f592d6fa7d in 838ms, sequenceid=490, compaction requested=true 2023-04-17 08:02:11,036 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:11,036 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:11,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:11,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:11,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:11,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:11,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:11,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:11,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:11,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b95a7bc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:11,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:11,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:11,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:11,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:11,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:11,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:11,088 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,088 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:11,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:11,089 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:11,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:11,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:11,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:11,089 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:11,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718591088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718591095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718591097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718591198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718591200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718591203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,267 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,267 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:11,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:11,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,268 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:11,268 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:11,268 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:11,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,268 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,368 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:11,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:11,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:11,370 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:11,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:11,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:11,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,379 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:11,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:11,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:11,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:11,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,380 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:11,381 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,381 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:11,381 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:11,381 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:11,381 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:11,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:11,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca23585[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:11,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:11,389 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:11,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:11,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:11,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:11,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:11,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:11,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:11,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:11,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:11,406 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718591403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,406 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718591403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718591410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:11,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,421 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,421 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:11,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/72f47771e78a4152b9444b17a5f7e185 2023-04-17 08:02:11,521 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:11,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:11,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:11,524 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:11,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:11,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:11,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,524 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,529 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:11,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:11,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:11,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:11,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:11,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:11,530 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:11,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:11,530 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:11,531 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:11,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:11,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:11,531 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:11,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:11,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aeb353[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:11,538 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:11,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,546 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:11,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:11,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:11,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:11,554 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:11,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:11,556 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:11,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:11,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:11,556 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:11,556 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:11,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:11,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:11,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,571 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,571 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:11,571 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:11,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,671 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:11,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:11,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:11,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:11,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:11,674 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,674 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:11,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:11,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:11,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:11,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:11,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:11,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:11,680 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,680 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:11,680 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:11,681 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:11,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:11,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:11,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:11,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:11,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618b509a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:11,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:11,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,696 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:11,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:11,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:11,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:11,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:11,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:11,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:11,706 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:11,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:11,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:11,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:11,706 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:11,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718591707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718591708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,714 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:11,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718591713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,734 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,735 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,734 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:11,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:11,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,735 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,735 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,735 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,835 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:11,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:11,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:11,838 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:11,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:11,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:11,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,846 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,846 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,846 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:11,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:11,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:11,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:11,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:11,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:11,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:11,847 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,847 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:11,847 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:11,848 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:11,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:11,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:11,848 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:11,854 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:11,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5795a350[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:11,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:11,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,862 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:11,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:11,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:11,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:11,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:11,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:11,872 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:11,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:11,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:11,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:11,872 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:11,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/bfe2c585176742018e65daf725fc2954 2023-04-17 08:02:11,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:11,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:11,881 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ccc0ee927ce44d4c9fe10306c9c66251 2023-04-17 08:02:11,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/72f47771e78a4152b9444b17a5f7e185 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185 2023-04-17 08:02:11,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:11,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:11,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:11,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:11,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:11,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:11,888 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:11,888 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:11,888 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185, entries=200, sequenceid=504, filesize=14.3 K 2023-04-17 08:02:11,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/bfe2c585176742018e65daf725fc2954 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954 2023-04-17 08:02:11,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954, entries=150, sequenceid=504, filesize=11.9 K 2023-04-17 08:02:11,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ccc0ee927ce44d4c9fe10306c9c66251 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251 2023-04-17 08:02:11,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251, entries=150, sequenceid=504, filesize=11.9 K 2023-04-17 08:02:11,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 614ac39c30038f7f3ad5b0f592d6fa7d in 835ms, sequenceid=504, compaction requested=true 2023-04-17 08:02:11,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:11,897 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:11,897 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:11,897 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:11,897 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:11,897 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:11,898 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51376 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:11,898 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:11,898 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:11,898 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:11,899 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,899 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,899 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/2ad46eac6b6b4cd6829d118bf56cc4f0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=50.2 K 2023-04-17 08:02:11,899 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/68bd7086f81f4b398e84645573bb1377, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=47.8 K 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2ad46eac6b6b4cd6829d118bf56cc4f0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527672 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 68bd7086f81f4b398e84645573bb1377, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527672 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e7b8f2adb23f44988f1825e5b2c289a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1681718528812 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fcc4874308b746549c9b8a733fddd73a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1681718528812 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting fe39986617594477b200a517d0b5d039, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681718528880 2023-04-17 08:02:11,900 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1ab3b6f177cd4b61aca87ebbd8e09140, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681718528880 2023-04-17 08:02:11,901 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 72f47771e78a4152b9444b17a5f7e185, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:11,901 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bfe2c585176742018e65daf725fc2954, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:11,906 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:11,908 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#411 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:11,924 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/c24aeb6d17db451397bd2deaea8aa931 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c24aeb6d17db451397bd2deaea8aa931 2023-04-17 08:02:11,927 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into c24aeb6d17db451397bd2deaea8aa931(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:11,928 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:11,928 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=12, startTime=1681718531897; duration=0sec 2023-04-17 08:02:11,928 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:11,928 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:11,929 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:11,929 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:11,929 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,929 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/271785deaa234a239119eb823a8c5b93, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=47.8 K 2023-04-17 08:02:11,929 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 271785deaa234a239119eb823a8c5b93, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681718527672 2023-04-17 08:02:11,930 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ba52643fb8a0443fbe43249ef07ddda2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1681718528812 2023-04-17 08:02:11,930 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f047e0b274534d5193f953429f33d818, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681718528880 2023-04-17 08:02:11,930 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ccc0ee927ce44d4c9fe10306c9c66251, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:11,935 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:11,988 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:11,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:11,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:11,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:11,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:11,991 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:11,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:11,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:11,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,991 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,996 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,996 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:11,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:11,996 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:11,996 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:11,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:11,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:11,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:11,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:11,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:11,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:11,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:11,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:11,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:11,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:11,997 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:02:11,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:11,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:11,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:11,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:11,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:11,998 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:12,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:12,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718592234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718592235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718592239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,324 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/b43b4a58522c4881b056b73c80e319fc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b43b4a58522c4881b056b73c80e319fc 2023-04-17 08:02:12,327 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into b43b4a58522c4881b056b73c80e319fc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:12,327 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:12,327 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=12, startTime=1681718531897; duration=0sec 2023-04-17 08:02:12,327 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:12,345 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/65f1ae5f851c4dae801d73f201ee2e78 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/65f1ae5f851c4dae801d73f201ee2e78 2023-04-17 08:02:12,348 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into 65f1ae5f851c4dae801d73f201ee2e78(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:12,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:12,348 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=12, startTime=1681718531897; duration=0sec 2023-04-17 08:02:12,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:12,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718592345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718592345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718592346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,409 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/bd74182a9dc2430bb7ce81380421a3e0 2023-04-17 08:02:12,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718592549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718592549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718592550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,816 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/7b7524818d644531bb79fd10e9f7bfb5 2023-04-17 08:02:12,826 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/7103de7f7bd44028a09e68d131e2cba8 2023-04-17 08:02:12,829 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/bd74182a9dc2430bb7ce81380421a3e0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0 2023-04-17 08:02:12,833 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0, entries=150, sequenceid=523, filesize=11.9 K 2023-04-17 08:02:12,834 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/7b7524818d644531bb79fd10e9f7bfb5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5 2023-04-17 08:02:12,837 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5, entries=150, sequenceid=523, filesize=11.9 K 2023-04-17 08:02:12,838 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/7103de7f7bd44028a09e68d131e2cba8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8 2023-04-17 08:02:12,840 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8, entries=150, sequenceid=523, filesize=11.9 K 2023-04-17 08:02:12,841 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 614ac39c30038f7f3ad5b0f592d6fa7d in 844ms, sequenceid=523, compaction requested=false 2023-04-17 08:02:12,841 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:12,841 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:12,841 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:12,841 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:12,841 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,841 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:12,841 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:12,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:12,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 08:02:12,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:12,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:12,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:12,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:12,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:12,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:12,871 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:12,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,871 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:12,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:12,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a36ba3a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:12,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:12,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,879 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:12,879 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:12,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:12,879 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:12,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718592875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718592876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718592880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:12,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:12,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:12,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:12,889 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:12,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:12,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:12,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:12,889 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:12,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:12,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:12,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:12,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:12,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:12,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:12,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,904 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:12,904 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:12,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:12,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:12,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718592981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718592981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:12,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:12,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718592986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,005 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,007 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,008 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:13,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:13,014 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:13,021 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,021 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:13,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7981ba04[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:13,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:13,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718593024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,028 DEBUG [Thread-2892] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:13,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:13,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:13,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:13,029 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:13,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718593029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,033 DEBUG [Thread-2888] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:13,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:13,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:13,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:13,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:13,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:13,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:13,039 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:13,039 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:13,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:13,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:13,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:13,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,054 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:13,055 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,155 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,157 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,157 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,177 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,177 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,178 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,178 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,179 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:13,179 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:13,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:13,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:13,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:13,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:13,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42353a2e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:13,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:13,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718593186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718593186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718593190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,196 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:13,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:13,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:13,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:13,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:13,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:13,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:13,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:13,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:13,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:13,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:13,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:13,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,214 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:13,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,221 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,221 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/9f16cd0059334b74a8c2fd065c229ce8 2023-04-17 08:02:13,321 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,324 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,325 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,329 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,331 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:13,331 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:13,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:13,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:13,331 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:13,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,338 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:13,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38408e1b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:13,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:13,339 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:13,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:13,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:13,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,346 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:13,354 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:13,354 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,354 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,354 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:13,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:13,356 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:13,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:13,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:13,356 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:13,356 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:13,362 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,362 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,363 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,364 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,362 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,371 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:13,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:13,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,371 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:13,371 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,471 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,473 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,474 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,480 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,480 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,480 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,480 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:13,480 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:13,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:13,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:13,481 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:13,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,487 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,488 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:13,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e80443[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:13,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:13,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:13,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:13,496 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:13,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:13,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45960 deadline: 1681718593495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718593496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:13,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718593497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:13,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,504 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:13,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:13,506 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:13,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:13,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:13,506 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:13,506 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:13,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,513 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,514 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,521 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:13,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:13,521 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,521 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,621 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,624 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,624 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,624 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,624 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,624 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,624 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,629 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,631 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,631 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,631 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:13,631 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:13,631 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:13,637 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,638 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,638 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:13,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cdc120[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:13,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:13,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,661 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,661 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:13,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:13,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:13,661 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:13,671 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:13,671 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,671 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,671 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:13,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:13,672 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:13,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:13,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:13,673 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:13,673 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:13,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/3eeffeaa2d5247fea834f1a0f7b17d25 2023-04-17 08:02:13,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,681 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/287039cbccf44e2981a59af2f71b81a6 2023-04-17 08:02:13,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/9f16cd0059334b74a8c2fd065c229ce8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8 2023-04-17 08:02:13,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:13,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:13,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:13,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:13,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:13,688 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:13,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:13,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:13,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,688 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:13,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8, entries=250, sequenceid=541, filesize=16.7 K 2023-04-17 08:02:13,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/3eeffeaa2d5247fea834f1a0f7b17d25 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25 2023-04-17 08:02:13,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25, entries=150, sequenceid=541, filesize=11.9 K 2023-04-17 08:02:13,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/287039cbccf44e2981a59af2f71b81a6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6 2023-04-17 08:02:13,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6, entries=150, sequenceid=541, filesize=11.9 K 2023-04-17 08:02:13,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 614ac39c30038f7f3ad5b0f592d6fa7d in 840ms, sequenceid=541, compaction requested=true 2023-04-17 08:02:13,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:13,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:13,696 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:13,697 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:13,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:13,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:13,697 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41916 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:13,697 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:13,697 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:13,697 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,697 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:13,697 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b43b4a58522c4881b056b73c80e319fc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=40.9 K 2023-04-17 08:02:13,697 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,698 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c24aeb6d17db451397bd2deaea8aa931, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=36.2 K 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b43b4a58522c4881b056b73c80e319fc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c24aeb6d17db451397bd2deaea8aa931, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting bd74182a9dc2430bb7ce81380421a3e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681718531086 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7b7524818d644531bb79fd10e9f7bfb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681718531086 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9f16cd0059334b74a8c2fd065c229ce8, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532216 2023-04-17 08:02:13,698 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3eeffeaa2d5247fea834f1a0f7b17d25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532226 2023-04-17 08:02:13,704 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:13,704 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:13,718 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/03a014e183674b65b60e7bc3b5055350 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/03a014e183674b65b60e7bc3b5055350 2023-04-17 08:02:13,728 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 03a014e183674b65b60e7bc3b5055350(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:13,728 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:13,728 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=13, startTime=1681718533696; duration=0sec 2023-04-17 08:02:13,728 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:13,728 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:13,729 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:13,729 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:13,730 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,730 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/65f1ae5f851c4dae801d73f201ee2e78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=36.2 K 2023-04-17 08:02:13,730 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 65f1ae5f851c4dae801d73f201ee2e78, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681718530955 2023-04-17 08:02:13,730 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7103de7f7bd44028a09e68d131e2cba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681718531086 2023-04-17 08:02:13,730 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 287039cbccf44e2981a59af2f71b81a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532226 2023-04-17 08:02:13,741 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:13,762 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/06b1e7600c1649da8c0f5edaaf16ff50 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/06b1e7600c1649da8c0f5edaaf16ff50 2023-04-17 08:02:13,767 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into 06b1e7600c1649da8c0f5edaaf16ff50(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:13,767 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:13,767 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=13, startTime=1681718533697; duration=0sec 2023-04-17 08:02:13,767 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:13,788 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:13,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:13,790 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:13,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:13,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:13,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,796 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:13,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:13,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:13,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:13,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:13,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:13,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:13,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:13,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:13,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:13,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:13,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:13,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:13,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:13,798 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:13,799 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:13,810 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/43917ad579864b8d8bc8601c4b7e8042 2023-04-17 08:02:13,822 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/05fe2738992f45159b6b59d287ed9ce7 2023-04-17 08:02:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:14,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:14,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718594026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,120 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/917a2cf95e614b6cb0a0ab2d1425dde9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/917a2cf95e614b6cb0a0ab2d1425dde9 2023-04-17 08:02:14,124 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into 917a2cf95e614b6cb0a0ab2d1425dde9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:14,124 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:14,124 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=13, startTime=1681718533696; duration=0sec 2023-04-17 08:02:14,124 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:14,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718594130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,238 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/868dbf364b3b4ad489bd709a765b49a3 2023-04-17 08:02:14,241 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/43917ad579864b8d8bc8601c4b7e8042 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042 2023-04-17 08:02:14,246 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042, entries=150, sequenceid=559, filesize=11.9 K 2023-04-17 08:02:14,246 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/05fe2738992f45159b6b59d287ed9ce7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7 2023-04-17 08:02:14,249 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7, entries=150, sequenceid=559, filesize=11.9 K 2023-04-17 08:02:14,250 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/868dbf364b3b4ad489bd709a765b49a3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3 2023-04-17 08:02:14,252 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3, entries=150, sequenceid=559, filesize=11.9 K 2023-04-17 08:02:14,253 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 614ac39c30038f7f3ad5b0f592d6fa7d in 455ms, sequenceid=559, compaction requested=false 2023-04-17 08:02:14,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:14,253 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:14,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:14,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:14,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:14,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:14,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68426f3d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:14,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:14,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:14,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:14,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:14,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:14,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:14,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:14,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:14,281 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:14,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:14,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:14,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:14,281 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:14,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:14,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,296 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:14,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,296 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,296 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:14,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:02:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:14,337 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:14,337 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:14,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/d428fabd254e41dca52b9395025ce3cf 2023-04-17 08:02:14,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45946 deadline: 1681718594355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,396 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:14,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:14,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:14,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:14,399 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:14,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:14,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:14,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,399 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:14,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:14,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:14,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:14,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:14,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:14,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:14,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:14,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:14,406 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:14,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:14,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:14,406 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:14,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:14,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544e0478[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:14,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:14,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:14,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:14,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:14,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:14,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:14,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,429 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:14,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:14,431 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:14,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:14,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:14,431 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:14,431 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:14,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:14,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:14,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:14,446 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,446 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,446 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,446 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718594464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,546 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:14,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:14,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:14,549 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:14,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:14,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:14,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,554 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:14,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:14,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:14,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:14,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:14,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:14,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:14,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:14,556 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:14,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:14,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:14,556 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:14,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,563 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:14,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b3d8ae[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:14,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:14,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,575 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:14,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:14,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:14,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,576 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:14,603 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:14,603 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,603 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:14,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:14,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:14,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:14,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:14,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:14,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:14,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,614 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:14,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,621 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,621 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,621 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718594667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,721 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:14,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:14,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:14,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:14,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:14,724 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:14,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:14,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:14,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:14,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:14,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:14,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:14,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:14,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:14,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:14,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:14,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:14,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,731 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:14,731 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:14,731 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,731 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:14,731 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:14,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,738 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:14,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1ad139[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:14,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:14,739 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:14,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:14,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,746 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:14,746 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:14,754 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:14,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,754 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/a317976a95024f7ea50adcf613046b26 2023-04-17 08:02:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:14,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:14,757 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:14,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:14,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:14,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:14,757 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:14,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,769 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:14,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:14,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,779 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:14,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,780 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:14,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,880 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:14,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:14,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:14,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:14,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:14,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:14,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:14,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:14,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:14,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:14,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:14,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:14,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:14,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:14,889 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:14,889 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:14,896 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,896 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:14,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755a6c40[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:14,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:14,897 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:14,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:14,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:14,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:14,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:14,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:14,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:14,914 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:14,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:14,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:14,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:14,914 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:14,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:14,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:14,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:14,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:14,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:14,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:14,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:14,929 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,929 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:14,929 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:14,930 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718594972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718594973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718594974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,029 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:15,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:15,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:15,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:15,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:15,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:15,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:15,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:15,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:15,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:15,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:15,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:15,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:15,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:15,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:15,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:15,039 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:15,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:15,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:15,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:15,040 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:15,040 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:15,046 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:15,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,046 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:15,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:15,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5113cb57[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:15,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:15,047 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,054 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:15,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,054 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:15,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:15,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:15,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:15,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:15,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:15,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:15,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:15,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:15,064 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:15,064 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:15,064 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:15,064 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:15,064 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:15,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:15,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:15,071 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:15,073 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:15,095 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:15,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:15,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:15,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:15,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:15,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:15,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,096 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:15,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:15,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,096 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:15,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:15,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:15,096 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/af7ad27c61b94c069a4a7cb8fd2729d3 2023-04-17 08:02:15,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/d428fabd254e41dca52b9395025ce3cf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf 2023-04-17 08:02:15,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf, entries=200, sequenceid=578, filesize=14.3 K 2023-04-17 08:02:15,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/a317976a95024f7ea50adcf613046b26 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26 2023-04-17 08:02:15,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26, entries=150, sequenceid=578, filesize=11.9 K 2023-04-17 08:02:15,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/af7ad27c61b94c069a4a7cb8fd2729d3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3 2023-04-17 08:02:15,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3, entries=150, sequenceid=578, filesize=11.9 K 2023-04-17 08:02:15,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 614ac39c30038f7f3ad5b0f592d6fa7d in 840ms, sequenceid=578, compaction requested=true 2023-04-17 08:02:15,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:15,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:15,177 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:15,177 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:15,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:15,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:15,177 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:15,177 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:15,178 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,178 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,178 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/03a014e183674b65b60e7bc3b5055350, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=38.7 K 2023-04-17 08:02:15,178 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/917a2cf95e614b6cb0a0ab2d1425dde9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=36.3 K 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 03a014e183674b65b60e7bc3b5055350, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532226 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 917a2cf95e614b6cb0a0ab2d1425dde9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532226 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 43917ad579864b8d8bc8601c4b7e8042, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681718532869 2023-04-17 08:02:15,178 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 05fe2738992f45159b6b59d287ed9ce7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681718532869 2023-04-17 08:02:15,179 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting d428fabd254e41dca52b9395025ce3cf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:15,179 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a317976a95024f7ea50adcf613046b26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:15,183 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#428 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:15,184 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:15,191 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/c2d5561471f545719610d8b73871723b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c2d5561471f545719610d8b73871723b 2023-04-17 08:02:15,193 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into c2d5561471f545719610d8b73871723b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:15,194 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:15,194 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=13, startTime=1681718535177; duration=0sec 2023-04-17 08:02:15,194 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:15,194 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:15,194 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:15,194 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:15,194 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,194 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/06b1e7600c1649da8c0f5edaaf16ff50, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=36.3 K 2023-04-17 08:02:15,195 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 06b1e7600c1649da8c0f5edaaf16ff50, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681718532226 2023-04-17 08:02:15,195 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 868dbf364b3b4ad489bd709a765b49a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681718532869 2023-04-17 08:02:15,195 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting af7ad27c61b94c069a4a7cb8fd2729d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:15,196 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:15,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:15,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:15,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:15,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:15,199 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:15,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:15,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:15,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:15,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,199 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,200 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:15,206 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/77dac2e62b8e40bbaba6de5f5ab08bda as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/77dac2e62b8e40bbaba6de5f5ab08bda 2023-04-17 08:02:15,209 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into 77dac2e62b8e40bbaba6de5f5ab08bda(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:15,209 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:15,209 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=13, startTime=1681718535177; duration=0sec 2023-04-17 08:02:15,209 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:15,211 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:15,211 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:15,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:15,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:15,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:15,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:15,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:15,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:15,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:15,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:15,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:15,213 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:15,213 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:15,213 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:15,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:15,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718595505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718595507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718595510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,590 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/85faf7b028b74f50ad459d0ae0138bde as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/85faf7b028b74f50ad459d0ae0138bde 2023-04-17 08:02:15,593 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 85faf7b028b74f50ad459d0ae0138bde(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:15,593 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:15,593 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=13, startTime=1681718535177; duration=0sec 2023-04-17 08:02:15,593 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718595612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718595614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718595614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,618 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e54e480edce74339b441afdb4b164d06 2023-04-17 08:02:15,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718595814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718595818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:15,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:15,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718595818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,026 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/983103d32d924a478ccea205eda81330 2023-04-17 08:02:16,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718596118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718596120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718596121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,433 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/cb3cdf2fdf88419c8965eb75c4ab724d 2023-04-17 08:02:16,436 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/e54e480edce74339b441afdb4b164d06 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06 2023-04-17 08:02:16,439 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06, entries=150, sequenceid=594, filesize=11.9 K 2023-04-17 08:02:16,440 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/983103d32d924a478ccea205eda81330 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330 2023-04-17 08:02:16,442 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330, entries=150, sequenceid=594, filesize=11.9 K 2023-04-17 08:02:16,443 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/cb3cdf2fdf88419c8965eb75c4ab724d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d 2023-04-17 08:02:16,447 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d, entries=150, sequenceid=594, filesize=11.9 K 2023-04-17 08:02:16,448 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 614ac39c30038f7f3ad5b0f592d6fa7d in 1235ms, sequenceid=594, compaction requested=false 2023-04-17 08:02:16,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:16,448 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:16,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:16,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:16,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:16,448 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:16,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:16,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:16,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:16,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c981f73[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:16,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:16,455 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:16,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:16,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:16,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:16,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:16,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:16,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:16,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:16,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:16,472 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:16,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:16,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:16,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:16,472 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:16,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:16,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:16,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:16,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:16,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:16,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:16,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:16,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:16,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:16,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:16,488 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:16,488 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:16,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:16,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:16,488 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:16,588 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:16,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:16,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:16,590 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:16,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:16,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:16,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:16,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,591 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:16,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,596 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:16,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:16,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:16,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:16,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:16,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:16,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:16,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:16,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:16,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:16,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:16,597 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:16,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:16,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:16,597 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 08:02:16,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:16,597 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:16,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:16,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:16,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:16,598 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:16,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:16,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:16,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718596639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718596639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718596640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718596741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718596748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718596748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718596945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718596949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:16,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:16,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718596950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,004 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/12963e1a7f374143ba2c88fc2077b6ec 2023-04-17 08:02:17,013 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/2449a0fc62a14fd1bd9f07e64b616639 2023-04-17 08:02:17,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45910 deadline: 1681718597031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,035 DEBUG [Thread-2892] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:17,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45954 deadline: 1681718597055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,060 DEBUG [Thread-2888] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:17,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718597249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718597254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718597256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,420 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b6092e3f855642289e3b079bbd800bc7 2023-04-17 08:02:17,423 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/12963e1a7f374143ba2c88fc2077b6ec as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec 2023-04-17 08:02:17,426 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec, entries=150, sequenceid=613, filesize=11.9 K 2023-04-17 08:02:17,427 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/2449a0fc62a14fd1bd9f07e64b616639 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639 2023-04-17 08:02:17,430 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639, entries=150, sequenceid=613, filesize=11.9 K 2023-04-17 08:02:17,430 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b6092e3f855642289e3b079bbd800bc7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7 2023-04-17 08:02:17,433 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7, entries=150, sequenceid=613, filesize=11.9 K 2023-04-17 08:02:17,433 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 614ac39c30038f7f3ad5b0f592d6fa7d in 836ms, sequenceid=613, compaction requested=true 2023-04-17 08:02:17,434 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:17,434 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:17,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:17,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:17,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:17,434 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:17,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:17,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:17,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:17,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d380c0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:17,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:17,464 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,471 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:17,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:17,471 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:17,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:17,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:17,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:17,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:17,479 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:17,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:17,481 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:17,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:17,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:17,481 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:17,481 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:17,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:17,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:17,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:17,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,495 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,495 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:17,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:17,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:17,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:17,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:17,496 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:17,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:17,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:17,496 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:17,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,596 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:17,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:17,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:17,600 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:17,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:17,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:17,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:17,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,600 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:17,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:17,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:17,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:17,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:17,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:17,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:17,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:17,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:17,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:17,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:17,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:17,605 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:17,605 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:17,605 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:17,605 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 08:02:17,605 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:17,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:17,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:17,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:17,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:17,606 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:17,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:17,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:17,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718597806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718597807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718597807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718597910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718597912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:17,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:17,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718597912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,013 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c 2023-04-17 08:02:18,021 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ad0bbceafe8b40d092134dca4882c22a 2023-04-17 08:02:18,029 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ee9f81d9063c4044a23b0164fe393538 2023-04-17 08:02:18,032 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c 2023-04-17 08:02:18,034 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c, entries=100, sequenceid=627, filesize=9.5 K 2023-04-17 08:02:18,035 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ad0bbceafe8b40d092134dca4882c22a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a 2023-04-17 08:02:18,037 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a, entries=100, sequenceid=627, filesize=9.5 K 2023-04-17 08:02:18,038 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/ee9f81d9063c4044a23b0164fe393538 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538 2023-04-17 08:02:18,040 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538, entries=100, sequenceid=627, filesize=9.5 K 2023-04-17 08:02:18,041 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 614ac39c30038f7f3ad5b0f592d6fa7d in 436ms, sequenceid=627, compaction requested=true 2023-04-17 08:02:18,041 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:18,041 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,041 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:18,041 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:18,041 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,041 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:18,041 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:18,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,071 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:18,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524dda76[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:18,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:18,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,079 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:18,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:18,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:18,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:18,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:18,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,087 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:18,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:18,089 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:18,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:18,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:18,089 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:18,089 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:18,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,097 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:18,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:18,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:18,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:18,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,123 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:18,123 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:18,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:18,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,122 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,123 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/dd4592ff2bfc451eba4a3fbc74e71df6 2023-04-17 08:02:18,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718598133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718598144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718598144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,223 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:18,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:18,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:18,225 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:18,225 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:18,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:18,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,226 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,238 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:18,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:18,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:18,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:18,238 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:18,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:18,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:18,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:18,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:18,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,246 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:18,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcbfa1b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:18,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:18,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718598245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:18,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:18,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:18,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:18,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718598252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718598252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:18,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:18,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:18,264 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:18,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:18,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:18,264 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:18,264 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:18,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:18,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,279 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:18,279 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:18,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:18,380 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:18,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:18,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:18,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:18,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:18,382 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:18,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:18,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:18,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:18,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:18,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:18,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:18,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:18,389 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,389 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:18,389 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:18,389 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:18,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:18,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,396 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:18,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c41d56[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:18,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:18,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:18,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:18,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:18,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:18,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:18,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:18,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:18,414 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:18,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:18,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:18,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:18,414 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:18,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,421 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,422 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:18,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,429 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:18,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,430 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:18,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,430 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718598450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718598456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718598456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,530 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:18,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:18,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:18,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:18,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:18,532 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:18,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:18,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:18,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,533 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:18,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:18,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:18,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:18,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:18,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:18,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:18,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:18,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:18,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:18,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:18,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:18,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:18,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/6ececbefdd354793aadba6cb72d84965 2023-04-17 08:02:18,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,546 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:18,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f9462e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:18,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:18,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f2c2b1c45b384cb2933a1293878d3bd7 2023-04-17 08:02:18,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/dd4592ff2bfc451eba4a3fbc74e71df6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6 2023-04-17 08:02:18,554 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:18,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:18,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:18,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:18,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:18,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:18,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:18,563 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:18,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:18,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:18,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:18,563 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:18,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6, entries=200, sequenceid=647, filesize=14.3 K 2023-04-17 08:02:18,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/6ececbefdd354793aadba6cb72d84965 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965 2023-04-17 08:02:18,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965, entries=150, sequenceid=647, filesize=11.9 K 2023-04-17 08:02:18,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f2c2b1c45b384cb2933a1293878d3bd7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7 2023-04-17 08:02:18,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,571 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,572 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,573 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:18,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:18,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:18,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:18,580 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:18,579 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:18,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,580 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:18,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:18,580 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7, entries=150, sequenceid=647, filesize=11.9 K 2023-04-17 08:02:18,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 614ac39c30038f7f3ad5b0f592d6fa7d in 466ms, sequenceid=647, compaction requested=true 2023-04-17 08:02:18,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:18,581 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:18,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:18,582 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:18,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:18,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:18,583 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:18,583 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:18,583 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,584 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c2d5561471f545719610d8b73871723b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=57.8 K 2023-04-17 08:02:18,585 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61608 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:18,585 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:18,585 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,585 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/85faf7b028b74f50ad459d0ae0138bde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=60.2 K 2023-04-17 08:02:18,586 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c2d5561471f545719610d8b73871723b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:18,586 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 85faf7b028b74f50ad459d0ae0138bde, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:18,586 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 983103d32d924a478ccea205eda81330, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681718534344 2023-04-17 08:02:18,586 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e54e480edce74339b441afdb4b164d06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681718534344 2023-04-17 08:02:18,587 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2449a0fc62a14fd1bd9f07e64b616639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681718535502 2023-04-17 08:02:18,587 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 12963e1a7f374143ba2c88fc2077b6ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681718535502 2023-04-17 08:02:18,587 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ad0bbceafe8b40d092134dca4882c22a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681718536633 2023-04-17 08:02:18,587 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ea6f2a4cbe8b4e54aae7126ad9b71c6c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681718536633 2023-04-17 08:02:18,587 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6ececbefdd354793aadba6cb72d84965, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:18,588 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting dd4592ff2bfc451eba4a3fbc74e71df6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:18,602 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:18,605 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:18,620 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/880809efde564dafb0a06a5cdd89e13e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/880809efde564dafb0a06a5cdd89e13e 2023-04-17 08:02:18,623 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 880809efde564dafb0a06a5cdd89e13e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:18,623 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:18,624 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=11, startTime=1681718538581; duration=0sec 2023-04-17 08:02:18,624 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:18,624 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:18,624 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/e1319174b2e44b31b75f9d1a4f5f3557 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1319174b2e44b31b75f9d1a4f5f3557 2023-04-17 08:02:18,625 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:18,625 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:18,625 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,625 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/77dac2e62b8e40bbaba6de5f5ab08bda, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=57.8 K 2023-04-17 08:02:18,625 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 77dac2e62b8e40bbaba6de5f5ab08bda, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681718534024 2023-04-17 08:02:18,626 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting cb3cdf2fdf88419c8965eb75c4ab724d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681718534344 2023-04-17 08:02:18,626 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b6092e3f855642289e3b079bbd800bc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681718535502 2023-04-17 08:02:18,626 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ee9f81d9063c4044a23b0164fe393538, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681718536633 2023-04-17 08:02:18,626 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f2c2b1c45b384cb2933a1293878d3bd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:18,628 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into e1319174b2e44b31b75f9d1a4f5f3557(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:18,628 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:18,628 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=11, startTime=1681718538582; duration=0sec 2023-04-17 08:02:18,628 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:18,633 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:18,680 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:18,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:18,682 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:18,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:18,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:18,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,683 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,706 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:18,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:18,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:18,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:18,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:18,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:18,707 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:18,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:18,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:18,708 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:18,708 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:18,708 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:18,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:18,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718598780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718598780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718598783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718598886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718598886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718598888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,040 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/3c8b8a80fcea40e3814ba343478786ff as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/3c8b8a80fcea40e3814ba343478786ff 2023-04-17 08:02:19,044 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into 3c8b8a80fcea40e3814ba343478786ff(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:19,044 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:19,044 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=11, startTime=1681718538582; duration=0sec 2023-04-17 08:02:19,044 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718599094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718599094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718599094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,115 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0 2023-04-17 08:02:19,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718599399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718599400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718599400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,524 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/71bcf3ceec984a118aa8dba05165fa73 2023-04-17 08:02:19,531 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/9db1eedca89d47ea8a4ea1871737ad69 2023-04-17 08:02:19,534 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0 2023-04-17 08:02:19,537 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0, entries=150, sequenceid=661, filesize=11.9 K 2023-04-17 08:02:19,537 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/71bcf3ceec984a118aa8dba05165fa73 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73 2023-04-17 08:02:19,540 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73, entries=150, sequenceid=661, filesize=11.9 K 2023-04-17 08:02:19,540 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/9db1eedca89d47ea8a4ea1871737ad69 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69 2023-04-17 08:02:19,543 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69, entries=150, sequenceid=661, filesize=11.9 K 2023-04-17 08:02:19,543 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 614ac39c30038f7f3ad5b0f592d6fa7d in 835ms, sequenceid=661, compaction requested=false 2023-04-17 08:02:19,544 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:19,544 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:19,544 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:19,544 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:19,544 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,544 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:19,544 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:19,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:19,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:19,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:19,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:19,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25dba10d[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:19,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:19,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:19,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:19,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:19,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:19,595 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:19,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:19,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:19,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:19,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:19,597 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:19,598 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:19,598 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:19,598 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:19,598 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:19,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:19,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:19,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,605 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:19,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,613 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:19,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:19,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:19,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:19,613 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:19,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:19,613 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:19,713 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:19,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:19,715 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:19,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:19,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:19,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:19,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:19,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:19,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:19,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:19,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:19,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:19,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:19,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:19,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:19,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:19,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:19,722 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:19,722 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:19,722 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:19,729 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/9aaf932105ef463ba54365e38f09e182 2023-04-17 08:02:19,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:19,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:19,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718599920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718599925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:19,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:19,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718599926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718600026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718600030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718600030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,138 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/570add4307684dc498d7155026882bef 2023-04-17 08:02:20,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718600231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718600233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718600233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718600537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718600537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:20,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718600537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,546 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8 2023-04-17 08:02:20,549 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/9aaf932105ef463ba54365e38f09e182 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182 2023-04-17 08:02:20,552 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182, entries=150, sequenceid=682, filesize=11.9 K 2023-04-17 08:02:20,552 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/570add4307684dc498d7155026882bef as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef 2023-04-17 08:02:20,555 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef, entries=150, sequenceid=682, filesize=11.9 K 2023-04-17 08:02:20,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8 2023-04-17 08:02:20,558 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8, entries=150, sequenceid=682, filesize=11.9 K 2023-04-17 08:02:20,558 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 614ac39c30038f7f3ad5b0f592d6fa7d in 836ms, sequenceid=682, compaction requested=true 2023-04-17 08:02:20,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:20,559 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:20,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:20,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:20,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:20,559 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:20,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:20,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,588 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:20,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:20,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67829c8f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:20,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:20,590 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:20,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:20,596 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:20,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,596 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:20,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:20,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:20,604 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:20,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:20,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:20,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:20,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:20,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:20,606 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:20,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:20,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:20,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:20,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,620 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:20,620 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,620 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:20,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:20,620 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:20,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:20,621 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,621 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:20,621 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,721 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:20,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:20,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:20,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:20,724 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:20,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:20,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:20,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:20,724 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,725 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:20,729 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:20,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:20,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:20,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:20,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:20,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:20,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:20,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:20,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:20,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:20,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:20,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:20,730 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:20,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:20,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:20,730 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:02:20,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:20,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:20,730 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:20,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:20,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:20,731 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:20,736 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/b2ca8de56edc4f93bba752f478a36701 2023-04-17 08:02:20,746 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/018aef715e934319981af96ad09458fc 2023-04-17 08:02:21,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:21,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:21,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718601065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718601068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718601069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,152 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/2e59926289724fbe8c63f617b1de59d5 2023-04-17 08:02:21,155 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/b2ca8de56edc4f93bba752f478a36701 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701 2023-04-17 08:02:21,159 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701, entries=150, sequenceid=694, filesize=11.9 K 2023-04-17 08:02:21,159 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/018aef715e934319981af96ad09458fc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc 2023-04-17 08:02:21,162 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc, entries=150, sequenceid=694, filesize=11.9 K 2023-04-17 08:02:21,162 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/2e59926289724fbe8c63f617b1de59d5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5 2023-04-17 08:02:21,165 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5, entries=150, sequenceid=694, filesize=11.9 K 2023-04-17 08:02:21,166 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 614ac39c30038f7f3ad5b0f592d6fa7d in 436ms, sequenceid=694, compaction requested=true 2023-04-17 08:02:21,166 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:21,166 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:21,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:21,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:21,166 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:21,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,171 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:21,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36fd1384[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:21,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:21,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:21,175 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:21,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:21,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718601188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718601194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718601195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,229 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:21,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:21,229 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:21,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:21,237 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:21,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,237 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,237 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:21,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:21,239 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:21,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:21,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:21,239 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:21,239 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:21,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,247 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:21,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,254 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,254 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:21,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718601297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718601297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718601301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,354 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:21,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:21,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:21,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:21,357 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:21,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:21,357 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:21,358 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,358 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,426 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,426 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,427 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,427 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:21,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:21,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:21,428 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,428 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:21,428 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:21,428 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:21,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:21,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,473 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,473 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:21,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6566eb43[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:21,475 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:21,475 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718601503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718601503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,507 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:21,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:21,507 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:21,507 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,507 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:21,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718601507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:21,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:21,521 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:21,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:21,523 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:21,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:21,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:21,523 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:21,523 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:21,529 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,529 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,529 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:21,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,538 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:21,538 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:21,538 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:21,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/eb7b8964125743f3b84ff454c94e5346 2023-04-17 08:02:21,638 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:21,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:21,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:21,641 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:21,641 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,641 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:21,641 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:21,641 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,641 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,645 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:21,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:21,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:21,647 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,647 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:21,647 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:21,647 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:21,647 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:21,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:21,655 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:21,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d4dfd2b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:21,655 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:21,662 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:21,662 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:21,662 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:21,670 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:21,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,670 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,670 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:21,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:21,672 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:21,672 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:21,672 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:21,672 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:21,672 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:21,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,679 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:21,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,687 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:21,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:21,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,688 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:21,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,688 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,788 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:21,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:21,790 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:21,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,790 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:21,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:21,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,791 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,795 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:21,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:21,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:21,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:21,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,796 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:21,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:21,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:21,797 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:21,797 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:21,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,804 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,804 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,804 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e586ce9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:21,805 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:21,805 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,812 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,812 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:21,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:21,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:21,812 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:21,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718601807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718601809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:21,820 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718601813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,820 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:21,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:21,822 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:21,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:21,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:21,822 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:21,822 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:21,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,829 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,829 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:21,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:21,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,837 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:21,837 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:21,837 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:21,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,838 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,938 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:21,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:21,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:21,940 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:21,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:21,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:21,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,940 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,945 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,945 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,946 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,946 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:21,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:21,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:21,947 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,947 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:21,947 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:21,947 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:21,947 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:21,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,954 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,954 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:21,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30e72395[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:21,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:21,955 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,962 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,962 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:21,962 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:21,962 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:21,962 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,962 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:21,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:21,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:21,972 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:21,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:21,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:21,972 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:21,972 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:21,979 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:21,979 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:21,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:21,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:21,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:21,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:21,987 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:21,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:21,988 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:21,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:21,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:21,988 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:21,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/bec480ec6c78488f9aa85718757ed9ce 2023-04-17 08:02:22,088 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:22,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:22,090 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:22,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:22,090 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:22,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,091 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,095 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,095 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,096 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:22,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:22,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:22,097 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,097 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:22,097 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:22,097 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:22,097 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:22,104 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:22,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6289ea39[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:22,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:22,105 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,112 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,112 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:22,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:22,112 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:22,120 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:22,120 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,120 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:22,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:22,124 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:22,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:22,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:22,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:22,124 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:22,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,129 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,130 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,138 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,138 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:22,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,138 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,238 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:22,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:22,240 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:22,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:22,240 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:22,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,241 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,245 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:22,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:22,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:22,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:22,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:22,247 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:22,247 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:22,254 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,254 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10dfe7e6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:22,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:22,255 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:22,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:22,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:22,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:22,270 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:22,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,270 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,271 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:22,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:22,272 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:22,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:22,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:22,272 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:22,272 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:22,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,279 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,287 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:22,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:22,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:22,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,288 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:22,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,288 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:22,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,288 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:22,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45946 deadline: 1681718602315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:22,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45960 deadline: 1681718602316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:22,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45926 deadline: 1681718602321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,388 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:22,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:22,390 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:22,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:22,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:22,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,395 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,396 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f40b950e2e654333bb85c8e0a132c106 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:22,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:22,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:22,397 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,397 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. started... 2023-04-17 08:02:22,397 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. as already flushing 2023-04-17 08:02:22,397 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:22,397 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:22,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/eb7b8964125743f3b84ff454c94e5346 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346 2023-04-17 08:02:22,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346, entries=200, sequenceid=715, filesize=14.3 K 2023-04-17 08:02:22,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/bec480ec6c78488f9aa85718757ed9ce as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce 2023-04-17 08:02:22,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:22,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dcdee30[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:22,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce, entries=150, sequenceid=715, filesize=11.9 K 2023-04-17 08:02:22,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:22,407 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/f40b950e2e654333bb85c8e0a132c106 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106 2023-04-17 08:02:22,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106, entries=150, sequenceid=715, filesize=11.9 K 2023-04-17 08:02:22,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 614ac39c30038f7f3ad5b0f592d6fa7d in 1236ms, sequenceid=715, compaction requested=true 2023-04-17 08:02:22,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:22,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:22,411 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:22,411 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:22,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:22,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:22,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:22,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:22,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:22,412 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,412 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:22,412 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64222 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/A is initiating minor compaction (all files) 2023-04-17 08:02:22,413 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/A in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,413 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/880809efde564dafb0a06a5cdd89e13e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=62.7 K 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/B is initiating minor compaction (all files) 2023-04-17 08:02:22,413 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/B in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 880809efde564dafb0a06a5cdd89e13e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:22,413 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1319174b2e44b31b75f9d1a4f5f3557, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=60.3 K 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 8f8a50bc9ead4d36a8dd84cb02dcb3a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681718538121 2023-04-17 08:02:22,413 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e1319174b2e44b31b75f9d1a4f5f3557, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 9aaf932105ef463ba54365e38f09e182, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681718538773 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 71bcf3ceec984a118aa8dba05165fa73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681718538121 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b2ca8de56edc4f93bba752f478a36701, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681718539917 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 570add4307684dc498d7155026882bef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681718538773 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting eb7b8964125743f3b84ff454c94e5346, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1681718541064 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 018aef715e934319981af96ad09458fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681718539917 2023-04-17 08:02:22,414 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bec480ec6c78488f9aa85718757ed9ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1681718541067 2023-04-17 08:02:22,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:22,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,421 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,421 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#B#compaction#458 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,421 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#A#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:22,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:22,423 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:22,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:22,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:22,423 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:22,423 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:22,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:22,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,449 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:22,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:22,450 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:22,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:22,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:22,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:22,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:22,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:22,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:22,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:22,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:22,463 DEBUG [Thread-2894] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:22,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:22,463 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:22,463 DEBUG [Thread-2894] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:22,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:22,527 DEBUG [Thread-2895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c240146 to 127.0.0.1:62477 2023-04-17 08:02:22,528 DEBUG [Thread-2895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:22,528 DEBUG [Thread-2897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d5f04c3 to 127.0.0.1:62477 2023-04-17 08:02:22,528 DEBUG [Thread-2897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:22,529 DEBUG [Thread-2899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6523d347 to 127.0.0.1:62477 2023-04-17 08:02:22,529 DEBUG [Thread-2899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:22,533 DEBUG [Thread-2903] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d90797d to 127.0.0.1:62477 2023-04-17 08:02:22,533 DEBUG [Thread-2903] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:22,533 DEBUG [Thread-2901] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28072326 to 127.0.0.1:62477 2023-04-17 08:02:22,533 DEBUG [Thread-2901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:22,563 DEBUG [Thread-2894] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:22,838 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/e1f9e9e14d1840fcb42cdaa7a47d7fb4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1f9e9e14d1840fcb42cdaa7a47d7fb4 2023-04-17 08:02:22,839 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/0c6e01e0d633453d983ad91e51854506 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0c6e01e0d633453d983ad91e51854506 2023-04-17 08:02:22,844 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/B of 614ac39c30038f7f3ad5b0f592d6fa7d into e1f9e9e14d1840fcb42cdaa7a47d7fb4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:22,844 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:22,844 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/B, priority=11, startTime=1681718542411; duration=0sec 2023-04-17 08:02:22,844 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:22,844 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/A of 614ac39c30038f7f3ad5b0f592d6fa7d into 0c6e01e0d633453d983ad91e51854506(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:22,844 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 08:02:22,844 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:22,844 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/A, priority=11, startTime=1681718542411; duration=0sec 2023-04-17 08:02:22,844 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:22,846 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 08:02:22,846 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 614ac39c30038f7f3ad5b0f592d6fa7d/C is initiating minor compaction (all files) 2023-04-17 08:02:22,846 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 614ac39c30038f7f3ad5b0f592d6fa7d/C in TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:22,846 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/3c8b8a80fcea40e3814ba343478786ff, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp, totalSize=60.3 K 2023-04-17 08:02:22,847 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3c8b8a80fcea40e3814ba343478786ff, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1681718537776 2023-04-17 08:02:22,847 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 9db1eedca89d47ea8a4ea1871737ad69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1681718538121 2023-04-17 08:02:22,847 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7f4d6e4ddaac4d408a2dfa6c90c04ed8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681718538773 2023-04-17 08:02:22,848 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2e59926289724fbe8c63f617b1de59d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681718539917 2023-04-17 08:02:22,848 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f40b950e2e654333bb85c8e0a132c106, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1681718541067 2023-04-17 08:02:22,858 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 614ac39c30038f7f3ad5b0f592d6fa7d#C#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:23,279 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/acf3f69667de4ac5986fe22c10508864 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/acf3f69667de4ac5986fe22c10508864 2023-04-17 08:02:23,284 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 614ac39c30038f7f3ad5b0f592d6fa7d/C of 614ac39c30038f7f3ad5b0f592d6fa7d into acf3f69667de4ac5986fe22c10508864(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:23,284 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:23,284 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d., storeName=614ac39c30038f7f3ad5b0f592d6fa7d/C, priority=11, startTime=1681718542411; duration=0sec 2023-04-17 08:02:23,284 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:23,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:02:23,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:23,321 DEBUG [Thread-2884] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x009b9721 to 127.0.0.1:62477 2023-04-17 08:02:23,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:23,321 DEBUG [Thread-2884] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:23,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:23,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:23,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:23,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:23,327 DEBUG [Thread-2886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a7ac4a1 to 127.0.0.1:62477 2023-04-17 08:02:23,327 DEBUG [Thread-2886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:23,334 DEBUG [Thread-2890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50d6899d to 127.0.0.1:62477 2023-04-17 08:02:23,334 DEBUG [Thread-2890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:23,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/8d2730b3d44b4bb5ae1ec7a95544aa1e 2023-04-17 08:02:24,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ac59532f6dbc4909a6aa6b4832b8b7af 2023-04-17 08:02:24,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b16b282e4f374fe49fa4584bfad9f8fb 2023-04-17 08:02:24,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/8d2730b3d44b4bb5ae1ec7a95544aa1e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8d2730b3d44b4bb5ae1ec7a95544aa1e 2023-04-17 08:02:24,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8d2730b3d44b4bb5ae1ec7a95544aa1e, entries=150, sequenceid=730, filesize=11.9 K 2023-04-17 08:02:24,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/ac59532f6dbc4909a6aa6b4832b8b7af as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ac59532f6dbc4909a6aa6b4832b8b7af 2023-04-17 08:02:24,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ac59532f6dbc4909a6aa6b4832b8b7af, entries=150, sequenceid=730, filesize=11.9 K 2023-04-17 08:02:24,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/b16b282e4f374fe49fa4584bfad9f8fb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b16b282e4f374fe49fa4584bfad9f8fb 2023-04-17 08:02:24,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b16b282e4f374fe49fa4584bfad9f8fb, entries=150, sequenceid=730, filesize=11.9 K 2023-04-17 08:02:24,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for 614ac39c30038f7f3ad5b0f592d6fa7d in 1275ms, sequenceid=730, compaction requested=false 2023-04-17 08:02:24,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:27,065 DEBUG [Thread-2888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x682c0b9e to 127.0.0.1:62477 2023-04-17 08:02:27,065 DEBUG [Thread-2888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:27,101 DEBUG [Thread-2892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6662e946 to 127.0.0.1:62477 2023-04-17 08:02:27,101 DEBUG [Thread-2892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 189 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 230 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-17 08:02:27,102 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2435 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7305 rows 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2433 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7293 rows 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2437 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7302 rows 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2414 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7242 rows 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(402): scanned 2431 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(403): verified 7293 rows 2023-04-17 08:02:27,103 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:02:27,103 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55409eb8 to 127.0.0.1:62477 2023-04-17 08:02:27,104 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:27,105 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:02:27,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:02:27,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:27,111 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718547111"}]},"ts":"1681718547111"} 2023-04-17 08:02:27,112 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:02:27,129 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:02:27,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, UNASSIGN}] 2023-04-17 08:02:27,131 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, UNASSIGN 2023-04-17 08:02:27,132 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=614ac39c30038f7f3ad5b0f592d6fa7d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:27,132 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718547132"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718547132"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718547132"}]},"ts":"1681718547132"} 2023-04-17 08:02:27,133 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:27,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:27,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:27,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 614ac39c30038f7f3ad5b0f592d6fa7d, disabling compactions & flushes 2023-04-17 08:02:27,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:27,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:27,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. after waiting 0 ms 2023-04-17 08:02:27,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:27,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 614ac39c30038f7f3ad5b0f592d6fa7d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 08:02:27,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=A 2023-04-17 08:02:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=B 2023-04-17 08:02:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 614ac39c30038f7f3ad5b0f592d6fa7d, store=C 2023-04-17 08:02:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:27,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:27,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/40841b5c5b894038957a7cc5b460dfb4 2023-04-17 08:02:27,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:28,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/b3ff28ec42544e70ba4131ebed76a135 2023-04-17 08:02:28,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/594b0c30cdee42b7b20560c4c7cf961f 2023-04-17 08:02:28,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/A/40841b5c5b894038957a7cc5b460dfb4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/40841b5c5b894038957a7cc5b460dfb4 2023-04-17 08:02:28,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/40841b5c5b894038957a7cc5b460dfb4, entries=150, sequenceid=737, filesize=11.9 K 2023-04-17 08:02:28,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/B/b3ff28ec42544e70ba4131ebed76a135 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/b3ff28ec42544e70ba4131ebed76a135 2023-04-17 08:02:28,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/b3ff28ec42544e70ba4131ebed76a135, entries=150, sequenceid=737, filesize=11.9 K 2023-04-17 08:02:28,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/.tmp/C/594b0c30cdee42b7b20560c4c7cf961f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/594b0c30cdee42b7b20560c4c7cf961f 2023-04-17 08:02:28,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/594b0c30cdee42b7b20560c4c7cf961f, entries=150, sequenceid=737, filesize=11.9 K 2023-04-17 08:02:28,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 614ac39c30038f7f3ad5b0f592d6fa7d in 889ms, sequenceid=737, compaction requested=true 2023-04-17 08:02:28,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/50bf93ba7dd848da8930c80129367b64, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/c41600e9bbf146dc92ee025223b29e90, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/2ad46eac6b6b4cd6829d118bf56cc4f0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b43b4a58522c4881b056b73c80e319fc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/03a014e183674b65b60e7bc3b5055350, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/85faf7b028b74f50ad459d0ae0138bde, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/880809efde564dafb0a06a5cdd89e13e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346] to archive 2023-04-17 08:02:28,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:28,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/a611a4584ad24236ba1bbe57d86b12b4 2023-04-17 08:02:28,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e5fb6ce60de6412ba7d4838dd9f6f6b0 2023-04-17 08:02:28,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/3b9aca90a839463c84a7cc182957a374 2023-04-17 08:02:28,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0747f42b53144d1eabfa1a483683391f 2023-04-17 08:02:28,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/50bf93ba7dd848da8930c80129367b64 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/50bf93ba7dd848da8930c80129367b64 2023-04-17 08:02:28,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/7436f5d3b97541e3a9793f4cd85e2739 2023-04-17 08:02:28,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fd466b9a98ae4e7e81a3c8de3951889d 2023-04-17 08:02:28,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/c41600e9bbf146dc92ee025223b29e90 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/c41600e9bbf146dc92ee025223b29e90 2023-04-17 08:02:28,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0678a21deb73400a890858dfa6245e9a 2023-04-17 08:02:28,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/604d58ee88b9447ab0e4cd152e5d444c 2023-04-17 08:02:28,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/2ad46eac6b6b4cd6829d118bf56cc4f0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/2ad46eac6b6b4cd6829d118bf56cc4f0 2023-04-17 08:02:28,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e7b8f2adb23f44988f1825e5b2c289a9 2023-04-17 08:02:28,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/fe39986617594477b200a517d0b5d039 2023-04-17 08:02:28,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/72f47771e78a4152b9444b17a5f7e185 2023-04-17 08:02:28,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b43b4a58522c4881b056b73c80e319fc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b43b4a58522c4881b056b73c80e319fc 2023-04-17 08:02:28,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/bd74182a9dc2430bb7ce81380421a3e0 2023-04-17 08:02:28,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9f16cd0059334b74a8c2fd065c229ce8 2023-04-17 08:02:28,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/03a014e183674b65b60e7bc3b5055350 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/03a014e183674b65b60e7bc3b5055350 2023-04-17 08:02:28,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/43917ad579864b8d8bc8601c4b7e8042 2023-04-17 08:02:28,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/d428fabd254e41dca52b9395025ce3cf 2023-04-17 08:02:28,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/85faf7b028b74f50ad459d0ae0138bde to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/85faf7b028b74f50ad459d0ae0138bde 2023-04-17 08:02:28,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/e54e480edce74339b441afdb4b164d06 2023-04-17 08:02:28,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/12963e1a7f374143ba2c88fc2077b6ec 2023-04-17 08:02:28,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/ea6f2a4cbe8b4e54aae7126ad9b71c6c 2023-04-17 08:02:28,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/dd4592ff2bfc451eba4a3fbc74e71df6 2023-04-17 08:02:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:28,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/880809efde564dafb0a06a5cdd89e13e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/880809efde564dafb0a06a5cdd89e13e 2023-04-17 08:02:28,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8f8a50bc9ead4d36a8dd84cb02dcb3a0 2023-04-17 08:02:28,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/9aaf932105ef463ba54365e38f09e182 2023-04-17 08:02:28,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/b2ca8de56edc4f93bba752f478a36701 2023-04-17 08:02:28,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/eb7b8964125743f3b84ff454c94e5346 2023-04-17 08:02:28,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/810cdad1e8cf40ebbac25a3fd94108e7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/453a21c4a461402ca81c8b89e0ec4089, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/68bd7086f81f4b398e84645573bb1377, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c24aeb6d17db451397bd2deaea8aa931, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/917a2cf95e614b6cb0a0ab2d1425dde9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c2d5561471f545719610d8b73871723b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1319174b2e44b31b75f9d1a4f5f3557, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce] to archive 2023-04-17 08:02:28,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:28,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad47735989f2431e8d4c3ec07571ce0e 2023-04-17 08:02:28,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e0c382cf693b42eb92ebfe8e356397bb 2023-04-17 08:02:28,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/cbf811d754184c8e83babd4eb432ed88 2023-04-17 08:02:28,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/810cdad1e8cf40ebbac25a3fd94108e7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/810cdad1e8cf40ebbac25a3fd94108e7 2023-04-17 08:02:28,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3b39cb8ece53438ca5c04c07ecb6d341 2023-04-17 08:02:28,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/14565b20c8d24528995e469ea9173652 2023-04-17 08:02:28,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/453a21c4a461402ca81c8b89e0ec4089 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/453a21c4a461402ca81c8b89e0ec4089 2023-04-17 08:02:28,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/d2c9144530214075a56f15e156d65c27 2023-04-17 08:02:28,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6a0a2c53615b425480f4a54f29cf3226 2023-04-17 08:02:28,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/68bd7086f81f4b398e84645573bb1377 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/68bd7086f81f4b398e84645573bb1377 2023-04-17 08:02:28,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1b08eaa1c74b4ddcb20d9e6ebe8c9333 2023-04-17 08:02:28,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/fcc4874308b746549c9b8a733fddd73a 2023-04-17 08:02:28,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/1ab3b6f177cd4b61aca87ebbd8e09140 2023-04-17 08:02:28,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c24aeb6d17db451397bd2deaea8aa931 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c24aeb6d17db451397bd2deaea8aa931 2023-04-17 08:02:28,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bfe2c585176742018e65daf725fc2954 2023-04-17 08:02:28,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/7b7524818d644531bb79fd10e9f7bfb5 2023-04-17 08:02:28,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/917a2cf95e614b6cb0a0ab2d1425dde9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/917a2cf95e614b6cb0a0ab2d1425dde9 2023-04-17 08:02:28,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/3eeffeaa2d5247fea834f1a0f7b17d25 2023-04-17 08:02:28,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/05fe2738992f45159b6b59d287ed9ce7 2023-04-17 08:02:28,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c2d5561471f545719610d8b73871723b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/c2d5561471f545719610d8b73871723b 2023-04-17 08:02:28,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/a317976a95024f7ea50adcf613046b26 2023-04-17 08:02:28,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/983103d32d924a478ccea205eda81330 2023-04-17 08:02:28,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/2449a0fc62a14fd1bd9f07e64b616639 2023-04-17 08:02:28,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ad0bbceafe8b40d092134dca4882c22a 2023-04-17 08:02:28,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1319174b2e44b31b75f9d1a4f5f3557 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1319174b2e44b31b75f9d1a4f5f3557 2023-04-17 08:02:28,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/6ececbefdd354793aadba6cb72d84965 2023-04-17 08:02:28,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/71bcf3ceec984a118aa8dba05165fa73 2023-04-17 08:02:28,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/570add4307684dc498d7155026882bef 2023-04-17 08:02:28,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/018aef715e934319981af96ad09458fc 2023-04-17 08:02:28,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/bec480ec6c78488f9aa85718757ed9ce 2023-04-17 08:02:28,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/e2f2d6b8f35348eabe8c277dd1182b71, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cee525fae05847079bc82d9336f40604, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/271785deaa234a239119eb823a8c5b93, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/65f1ae5f851c4dae801d73f201ee2e78, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/06b1e7600c1649da8c0f5edaaf16ff50, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/77dac2e62b8e40bbaba6de5f5ab08bda, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/3c8b8a80fcea40e3814ba343478786ff, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106] to archive 2023-04-17 08:02:28,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:28,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8808ed3a6a504260be29ce718ae7a35b 2023-04-17 08:02:28,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b7ad3e7ae03845c58e9b28c4bae3d9d7 2023-04-17 08:02:28,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/dfc1d0fb827e4fdfaa759341b26cf6b3 2023-04-17 08:02:28,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/e2f2d6b8f35348eabe8c277dd1182b71 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/e2f2d6b8f35348eabe8c277dd1182b71 2023-04-17 08:02:28,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cc22b86b172241e795ade69d0044525f 2023-04-17 08:02:28,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/aff9d448c99645a9b64aa813b033b012 2023-04-17 08:02:28,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cee525fae05847079bc82d9336f40604 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cee525fae05847079bc82d9336f40604 2023-04-17 08:02:28,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/49e05a9b293a4283b95d148c7e74146c 2023-04-17 08:02:28,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/26c63ef398664a508a446abb1906c94b 2023-04-17 08:02:28,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/271785deaa234a239119eb823a8c5b93 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/271785deaa234a239119eb823a8c5b93 2023-04-17 08:02:28,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/8500043867e04a7cb290eaab39ececac 2023-04-17 08:02:28,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ba52643fb8a0443fbe43249ef07ddda2 2023-04-17 08:02:28,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f047e0b274534d5193f953429f33d818 2023-04-17 08:02:28,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/65f1ae5f851c4dae801d73f201ee2e78 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/65f1ae5f851c4dae801d73f201ee2e78 2023-04-17 08:02:28,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ccc0ee927ce44d4c9fe10306c9c66251 2023-04-17 08:02:28,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7103de7f7bd44028a09e68d131e2cba8 2023-04-17 08:02:28,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/06b1e7600c1649da8c0f5edaaf16ff50 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/06b1e7600c1649da8c0f5edaaf16ff50 2023-04-17 08:02:28,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/287039cbccf44e2981a59af2f71b81a6 2023-04-17 08:02:28,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/868dbf364b3b4ad489bd709a765b49a3 2023-04-17 08:02:28,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/77dac2e62b8e40bbaba6de5f5ab08bda to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/77dac2e62b8e40bbaba6de5f5ab08bda 2023-04-17 08:02:28,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/af7ad27c61b94c069a4a7cb8fd2729d3 2023-04-17 08:02:28,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/cb3cdf2fdf88419c8965eb75c4ab724d 2023-04-17 08:02:28,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b6092e3f855642289e3b079bbd800bc7 2023-04-17 08:02:28,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/ee9f81d9063c4044a23b0164fe393538 2023-04-17 08:02:28,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/3c8b8a80fcea40e3814ba343478786ff to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/3c8b8a80fcea40e3814ba343478786ff 2023-04-17 08:02:28,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f2c2b1c45b384cb2933a1293878d3bd7 2023-04-17 08:02:28,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/9db1eedca89d47ea8a4ea1871737ad69 2023-04-17 08:02:28,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/7f4d6e4ddaac4d408a2dfa6c90c04ed8 2023-04-17 08:02:28,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/2e59926289724fbe8c63f617b1de59d5 2023-04-17 08:02:28,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/f40b950e2e654333bb85c8e0a132c106 2023-04-17 08:02:28,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/recovered.edits/740.seqid, newMaxSeqId=740, maxSeqId=1 2023-04-17 08:02:28,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d. 2023-04-17 08:02:28,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 614ac39c30038f7f3ad5b0f592d6fa7d: 2023-04-17 08:02:28,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:28,288 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=614ac39c30038f7f3ad5b0f592d6fa7d, regionState=CLOSED 2023-04-17 08:02:28,289 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718548288"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718548288"}]},"ts":"1681718548288"} 2023-04-17 08:02:28,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-17 08:02:28,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 614ac39c30038f7f3ad5b0f592d6fa7d, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.1560 sec 2023-04-17 08:02:28,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-17 08:02:28,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=614ac39c30038f7f3ad5b0f592d6fa7d, UNASSIGN in 1.1610 sec 2023-04-17 08:02:28,292 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718548292"}]},"ts":"1681718548292"} 2023-04-17 08:02:28,293 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:02:28,320 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:02:28,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2150 sec 2023-04-17 08:02:29,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 08:02:29,219 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-17 08:02:29,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:02:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,224 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,225 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 08:02:29,234 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:29,238 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/recovered.edits] 2023-04-17 08:02:29,244 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0c6e01e0d633453d983ad91e51854506 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/0c6e01e0d633453d983ad91e51854506 2023-04-17 08:02:29,246 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/40841b5c5b894038957a7cc5b460dfb4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/40841b5c5b894038957a7cc5b460dfb4 2023-04-17 08:02:29,248 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8d2730b3d44b4bb5ae1ec7a95544aa1e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/A/8d2730b3d44b4bb5ae1ec7a95544aa1e 2023-04-17 08:02:29,251 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ac59532f6dbc4909a6aa6b4832b8b7af to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/ac59532f6dbc4909a6aa6b4832b8b7af 2023-04-17 08:02:29,253 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/b3ff28ec42544e70ba4131ebed76a135 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/b3ff28ec42544e70ba4131ebed76a135 2023-04-17 08:02:29,254 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1f9e9e14d1840fcb42cdaa7a47d7fb4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/B/e1f9e9e14d1840fcb42cdaa7a47d7fb4 2023-04-17 08:02:29,258 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/594b0c30cdee42b7b20560c4c7cf961f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/594b0c30cdee42b7b20560c4c7cf961f 2023-04-17 08:02:29,259 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/acf3f69667de4ac5986fe22c10508864 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/acf3f69667de4ac5986fe22c10508864 2023-04-17 08:02:29,261 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b16b282e4f374fe49fa4584bfad9f8fb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/C/b16b282e4f374fe49fa4584bfad9f8fb 2023-04-17 08:02:29,266 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/recovered.edits/740.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d/recovered.edits/740.seqid 2023-04-17 08:02:29,267 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/614ac39c30038f7f3ad5b0f592d6fa7d 2023-04-17 08:02:29,267 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:02:29,271 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,272 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:02:29,274 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:02:29,275 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,275 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:02:29,275 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718549275"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:29,276 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:02:29,276 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 614ac39c30038f7f3ad5b0f592d6fa7d, NAME => 'TestAcidGuarantees,,1681718520245.614ac39c30038f7f3ad5b0f592d6fa7d.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:02:29,276 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:02:29,276 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718549276"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:29,278 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:02:29,313 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,315 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 93 msec 2023-04-17 08:02:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 08:02:29,327 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-17 08:02:29,341 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 226) - Thread LEAK? -, OpenFileDescriptor=535 (was 503) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=274 (was 281), ProcessCount=167 (was 167), AvailableMemoryMB=7694 (was 7638) - AvailableMemoryMB LEAK? - 2023-04-17 08:02:29,353 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=535, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=167, AvailableMemoryMB=7694 2023-04-17 08:02:29,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:02:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:02:29,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:29,358 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 08:02:29,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-17 08:02:29,359 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 08:02:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:29,360 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:29,361 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 empty. 2023-04-17 08:02:29,362 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:29,362 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:02:29,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:29,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:29,775 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:02:29,778 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4972af4c15665fa4c6eba818ffa6cdb1, NAME => 'TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp 2023-04-17 08:02:29,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:30,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:02:30,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4972af4c15665fa4c6eba818ffa6cdb1, disabling compactions & flushes 2023-04-17 08:02:30,191 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. after waiting 0 ms 2023-04-17 08:02:30,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,191 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:30,197 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 08:02:30,199 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718550199"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718550199"}]},"ts":"1681718550199"} 2023-04-17 08:02:30,201 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 08:02:30,202 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 08:02:30,202 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718550202"}]},"ts":"1681718550202"} 2023-04-17 08:02:30,204 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 08:02:30,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, ASSIGN}] 2023-04-17 08:02:30,248 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, ASSIGN 2023-04-17 08:02:30,250 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=false 2023-04-17 08:02:30,402 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:30,403 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718550402"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718550402"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718550402"}]},"ts":"1681718550402"} 2023-04-17 08:02:30,405 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:30,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:30,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4972af4c15665fa4c6eba818ffa6cdb1, NAME => 'TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:02:30,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:02:30,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,566 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,569 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A 2023-04-17 08:02:30,569 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A 2023-04-17 08:02:30,570 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:30,570 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName A 2023-04-17 08:02:30,571 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:30,571 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,574 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B 2023-04-17 08:02:30,574 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B 2023-04-17 08:02:30,575 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:30,575 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName B 2023-04-17 08:02:30,576 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:30,576 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,579 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C 2023-04-17 08:02:30,579 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C 2023-04-17 08:02:30,579 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:30,579 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName C 2023-04-17 08:02:30,580 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:30,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:02:30,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:30,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 08:02:30,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 4972af4c15665fa4c6eba818ffa6cdb1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62611100, jitterRate=-0.06702190637588501}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:02:30,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:30,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., pid=56, masterSystemTime=1681718550557 2023-04-17 08:02:30,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:30,590 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:30,591 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718550590"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718550590"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718550590"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718550590"}]},"ts":"1681718550590"} 2023-04-17 08:02:30,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-17 08:02:30,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 in 186 msec 2023-04-17 08:02:30,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-17 08:02:30,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, ASSIGN in 347 msec 2023-04-17 08:02:30,595 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 08:02:30,595 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718550595"}]},"ts":"1681718550595"} 2023-04-17 08:02:30,596 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 08:02:30,604 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 08:02:30,606 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2480 sec 2023-04-17 08:02:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 08:02:31,469 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-17 08:02:31,470 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x0c217430 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:31,522 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ae8f5b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:31,526 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:31,528 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:31,530 DEBUG [Listener at localhost.localdomain/40407] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 08:02:31,533 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 08:02:31,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 08:02:31,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 08:02:31,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:31,949 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:02:31,952 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 08:02:31,953 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 08:02:31,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 08:02:31,962 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, REOPEN/MOVE}] 2023-04-17 08:02:31,963 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, REOPEN/MOVE 2023-04-17 08:02:31,964 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:31,964 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718551964"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718551964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718551964"}]},"ts":"1681718551964"} 2023-04-17 08:02:31,965 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:32,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 4972af4c15665fa4c6eba818ffa6cdb1, disabling compactions & flushes 2023-04-17 08:02:32,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. after waiting 0 ms 2023-04-17 08:02:32,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 08:02:32,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:32,126 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4972af4c15665fa4c6eba818ffa6cdb1 to self. 2023-04-17 08:02:32,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,128 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=CLOSED 2023-04-17 08:02:32,128 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718552128"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718552128"}]},"ts":"1681718552128"} 2023-04-17 08:02:32,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-17 08:02:32,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 in 165 msec 2023-04-17 08:02:32,132 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,39843,1681718384756; forceNewPlan=false, retain=true 2023-04-17 08:02:32,283 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,283 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718552283"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718552283"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718552283"}]},"ts":"1681718552283"} 2023-04-17 08:02:32,286 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:32,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4972af4c15665fa4c6eba818ffa6cdb1, NAME => 'TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.', STARTKEY => '', ENDKEY => ''} 2023-04-17 08:02:32,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 08:02:32,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,450 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,452 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A 2023-04-17 08:02:32,452 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A 2023-04-17 08:02:32,452 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:32,452 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName A 2023-04-17 08:02:32,453 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:32,454 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,454 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B 2023-04-17 08:02:32,454 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B 2023-04-17 08:02:32,454 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:32,455 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName B 2023-04-17 08:02:32,455 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:32,455 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,456 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C 2023-04-17 08:02:32,456 DEBUG [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C 2023-04-17 08:02:32,456 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 08:02:32,456 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4972af4c15665fa4c6eba818ffa6cdb1 columnFamilyName C 2023-04-17 08:02:32,456 INFO [StoreOpener-4972af4c15665fa4c6eba818ffa6cdb1-1] regionserver.HStore(310): Store=4972af4c15665fa4c6eba818ffa6cdb1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 08:02:32,456 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 08:02:32,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 4972af4c15665fa4c6eba818ffa6cdb1; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70537933, jitterRate=0.05109710991382599}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 08:02:32,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:32,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., pid=61, masterSystemTime=1681718552440 2023-04-17 08:02:32,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,462 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,462 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718552462"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681718552462"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681718552462"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681718552462"}]},"ts":"1681718552462"} 2023-04-17 08:02:32,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-17 08:02:32,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 in 177 msec 2023-04-17 08:02:32,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-17 08:02:32,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, REOPEN/MOVE in 502 msec 2023-04-17 08:02:32,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-17 08:02:32,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 508 msec 2023-04-17 08:02:32,469 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 933 msec 2023-04-17 08:02:32,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-17 08:02:32,471 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5c51ec to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,504 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ddb3e63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,505 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x5be55982 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,513 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@630ea023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,513 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x2f11e84a to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,521 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec98d63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,521 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3a1010a6 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,529 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ae3ec16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,530 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x4143b9ae to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,546 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bec0400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,547 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x34b8bf14 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,563 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f6f7187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,563 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x387217bb to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,571 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c34fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,572 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x7c00a18c to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,588 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5467be78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,588 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x3da68fe0 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,605 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44535519, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,605 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(139): Connect 0x7fd90ac8 to 127.0.0.1:62477 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 08:02:32,621 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7689435c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 08:02:32,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:32,631 DEBUG [hconnection-0x113b13bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,632 DEBUG [hconnection-0x2118a809-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,633 DEBUG [hconnection-0x6359bd3b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,636 DEBUG [hconnection-0x6f2b5fde-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,637 DEBUG [hconnection-0x3e06955e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,637 DEBUG [hconnection-0x2ee8f836-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,638 DEBUG [hconnection-0x6284b90f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,638 DEBUG [hconnection-0x48485853-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,638 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,638 DEBUG [hconnection-0x3a934cac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,638 DEBUG [hconnection-0x5fffab70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 08:02:32,639 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,642 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,642 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,642 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,642 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,642 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,644 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,650 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 08:02:32,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:32,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:32,650 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:32,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:32,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:32,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,650 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,653 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:32,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:32,654 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,654 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:32,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:32,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:32,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:32,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,654 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:32,655 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:32,655 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:32,655 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,655 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:32,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:32,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 08:02:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:32,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:32,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:32,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:32,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:32,658 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:32,659 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,659 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:32,659 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:32,659 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,659 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:32,659 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:32,669 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,669 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,669 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,669 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:32,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343f8997[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:32,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:32,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718612669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718612669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718612670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718612671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718612671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,678 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,679 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:32,679 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:32,679 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:32,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,679 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:32,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:32,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:32,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:32,689 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:32,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:32,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:32,689 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:32,689 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:32,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,695 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:32,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:32,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:32,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,704 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:32,704 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:32,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718612773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718612773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718612773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718612773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718612774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,804 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:32,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:32,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:32,807 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,807 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:32,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:32,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,821 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:32,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:32,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:32,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:32,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,821 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:32,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:32,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:32,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:32,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:32,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:32,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ca650b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:32,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:32,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:32,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:32,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:32,837 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,837 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:32,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:32,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:32,860 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:32,860 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:32,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:32,862 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:32,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:32,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:32,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:32,862 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:32,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,870 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:32,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:32,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:32,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:32,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:32,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:32,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:32,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,879 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:32,879 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:32,879 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:32,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:32,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718612975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718612975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718612976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718612976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:32,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718612976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,979 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:32,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:32,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:32,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:32,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:32,982 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:32,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:32,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:32,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,982 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:32,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:32,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:32,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:32,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:32,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:32,988 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,988 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:32,988 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:32,988 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:32,988 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:32,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:32,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:32,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:32,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:32,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:32,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76454103[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:32,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:32,997 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,003 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,014 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,014 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,021 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:33,028 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,029 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,029 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,029 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,070 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304176faf167d5cb6424b86b56bfbb704958c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176faf167d5cb6424b86b56bfbb704958c 2023-04-17 08:02:33,074 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1720d68a0c974517823ce24255d4a3f9 2023-04-17 08:02:33,129 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,132 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,132 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,137 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,138 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,140 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,140 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,140 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:33,140 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:33,140 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:33,145 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:33,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@242b8d95[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:33,146 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:33,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,162 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,163 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,164 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,164 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,171 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:33,179 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,179 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,179 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,279 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718613279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718613279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718613279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718613279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718613279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,281 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,281 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,282 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,287 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,287 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,287 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:33,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,288 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:33,289 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:33,289 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:33,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,295 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,296 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:33,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c10cbb6[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:33,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:33,296 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,303 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,304 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,304 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,304 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,312 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,314 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,314 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,314 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,335 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,335 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,336 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,345 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,346 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,346 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,446 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,452 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,452 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:33,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:33,463 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:33,463 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:33,464 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:33,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,470 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26381902[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:33,471 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:33,472 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,478 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,478 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,479 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,479 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/bd3cb2a7002741739dd9a423fb346376 2023-04-17 08:02:33,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,487 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,487 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,487 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,489 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,489 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,489 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,495 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,495 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,495 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,495 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,503 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,503 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,504 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:33,504 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,504 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,604 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,607 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,607 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:33,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,613 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:33,613 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:33,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:33,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:33,614 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:33,620 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,620 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:33,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aa9199f[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:33,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:33,622 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,628 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,629 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,629 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,637 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,639 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,639 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,639 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,646 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,647 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:33,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,653 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,654 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,654 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,654 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,754 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,756 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,757 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,762 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,762 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,762 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:33,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:33,763 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:33,763 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:33,764 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:33,770 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,770 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:33,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296747a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:33,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:33,771 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,778 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,778 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:33,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:33,779 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:33,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:33,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718613782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718613783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718613783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718613784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:33,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718613784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:33,787 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:33,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:33,789 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:33,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:33,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:33,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:33,789 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:33,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:33,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:33,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,796 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,797 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:33,810 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 08:02:33,819 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,819 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:33,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:33,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:33,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:33,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:33,821 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:33,821 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:33,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/0b98c1a06b224ca8a92fe58d52629329 2023-04-17 08:02:33,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1720d68a0c974517823ce24255d4a3f9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9 2023-04-17 08:02:33,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9, entries=150, sequenceid=16, filesize=25.1 K 2023-04-17 08:02:33,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/bd3cb2a7002741739dd9a423fb346376 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376 2023-04-17 08:02:33,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376, entries=150, sequenceid=16, filesize=11.6 K 2023-04-17 08:02:33,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/0b98c1a06b224ca8a92fe58d52629329 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329 2023-04-17 08:02:33,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329, entries=150, sequenceid=16, filesize=11.6 K 2023-04-17 08:02:33,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1247ms, sequenceid=16, compaction requested=false 2023-04-17 08:02:33,904 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 08:02:33,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:33,921 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:33,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:33,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:33,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:33,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:33,924 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:33,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:33,924 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:33,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,925 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:33,928 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:33,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:33,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:33,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:33,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:33,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:33,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:33,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:33,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:33,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:33,930 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:33,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:33,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:33,930 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 08:02:33,930 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:33,931 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:33,931 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:33,931 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:33,931 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:33,931 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:33,949 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304175c61546012c04fe4ae97d74d0b9ed6f9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175c61546012c04fe4ae97d74d0b9ed6f9 2023-04-17 08:02:33,953 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/e693b4c1373345dabdb7bb17d33d6ea2 2023-04-17 08:02:34,362 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e710c5b696d249acaac6c6f4a9b64efb 2023-04-17 08:02:34,369 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/ba4d5a8cfa404eb5b8bf1f3814b2f661 2023-04-17 08:02:34,371 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/e693b4c1373345dabdb7bb17d33d6ea2 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2 2023-04-17 08:02:34,374 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2, entries=150, sequenceid=36, filesize=25.1 K 2023-04-17 08:02:34,374 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e710c5b696d249acaac6c6f4a9b64efb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb 2023-04-17 08:02:34,378 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 08:02:34,378 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/ba4d5a8cfa404eb5b8bf1f3814b2f661 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661 2023-04-17 08:02:34,382 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 08:02:34,382 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 4972af4c15665fa4c6eba818ffa6cdb1 in 452ms, sequenceid=36, compaction requested=false 2023-04-17 08:02:34,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:34,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:34,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:34,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:34,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:34,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,413 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:34,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683142e5[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:34,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:34,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:34,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:34,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:34,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,429 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:34,443 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:34,443 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,443 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,443 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:34,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:34,445 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:34,445 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:34,445 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:34,445 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:34,445 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:34,453 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,453 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,454 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,454 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,456 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,462 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:34,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,462 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:34,462 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,462 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,562 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:34,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:34,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:34,565 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:34,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:34,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:34,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,565 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,570 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,570 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,571 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,571 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:34,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:34,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:34,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:34,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:34,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,572 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:34,574 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,574 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:34,574 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:34,574 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,574 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:34,575 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:34,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,578 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,579 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:34,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ea7a6e8[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:34,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:34,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:34,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:34,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:34,587 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:34,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,595 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,595 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,595 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:34,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:34,597 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:34,597 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:34,597 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:34,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,604 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,604 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,606 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:34,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:34,612 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,613 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:34,613 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:34,713 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:34,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:34,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:34,716 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:34,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:34,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:34,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,716 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:34,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:34,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:34,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:34,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,723 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:34,724 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,724 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:34,724 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:34,724 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:34,724 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:34,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,728 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,729 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:34,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf03342[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:34,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:34,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,737 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:34,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:34,737 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:34,737 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:34,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:34,745 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,745 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:34,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:34,747 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:34,748 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:34,748 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:34,753 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,753 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,753 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,754 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,756 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,778 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:34,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,780 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:34,780 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:34,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:34,818 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:34,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718554836, max timestamp=1681718554842Num uniques 859; 2023-04-17 08:02:34,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718554836, max timestamp=1681718554842Num uniques 859; 2023-04-17 08:02:34,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:34,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:34,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718554836, max timestamp=1681718554842Num uniques 859; 2023-04-17 08:02:34,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718614856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718614857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718614857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718614861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718614861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,880 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:34,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:34,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:34,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:34,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:34,882 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:34,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:34,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:34,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,882 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:34,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:34,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:34,897 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,897 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:34,897 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:34,897 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:34,897 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:34,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,904 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d74b502[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:34,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:34,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:34,912 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,912 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:34,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:34,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:34,920 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:34,920 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,920 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:34,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:34,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:34,922 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:34,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:34,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:34,922 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:34,923 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:34,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,929 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:34,930 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:34,937 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:34,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:34,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:34,937 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:34,937 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:34,937 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:34,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:34,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718614962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718614962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718614962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718614964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:34,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:34,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718614964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,037 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,040 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,040 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,042 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,042 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,045 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:35,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,047 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,047 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,053 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f30f823[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,072 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,072 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,079 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,087 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,087 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,087 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,088 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718615165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718615165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718615166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718615167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718615168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,187 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,190 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,190 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,192 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:35,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,196 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,197 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,203 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c1383cb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,222 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,222 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,228 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,228 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,228 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,229 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,230 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,236 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304174e9906b541a74fdba20074b57e4797ff to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e9906b541a74fdba20074b57e4797ff 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,238 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,238 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:35,338 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,341 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,341 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:35,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,346 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,371 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,372 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,372 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6144d128[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,387 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,397 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,397 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,397 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,405 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,413 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,413 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,413 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718615468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718615469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718615469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718615470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718615472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,513 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,516 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,516 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,520 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,520 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,520 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,520 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,521 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,528 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,528 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4930b4a7[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,547 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,547 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,562 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,562 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,641 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1195567c76ae447183387e45f4884ec9 2023-04-17 08:02:35,662 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,665 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,666 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,670 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,670 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:35,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,672 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,672 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,672 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,672 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,672 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,678 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,678 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,679 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,679 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2129ac42[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,687 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,697 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,697 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,697 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,705 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,712 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:35,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,712 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,712 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,812 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:35,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:35,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:35,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:35,815 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:35,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:35,815 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:35,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,816 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,820 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,820 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:35,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:35,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:35,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:35,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:35,822 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:35,822 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:35,828 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,828 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,829 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:35,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61646e77[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:35,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:35,830 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,855 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:35,855 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:35,856 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,856 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:35,856 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:35,886 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,886 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,886 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,886 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:35,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:35,888 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:35,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:35,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:35,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:35,888 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:35,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:35,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,904 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,905 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:35,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:35,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:35,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,912 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:35,912 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:35,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,912 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:35,912 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:35,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718615971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718615972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718615973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718615973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:35,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:35,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718615987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,012 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:36,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:36,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:36,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:36,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:36,015 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:36,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:36,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:36,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,016 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,023 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:36,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:36,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:36,025 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,025 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:36,025 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:36,025 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:36,025 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:36,036 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,036 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:36,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266e2ee2[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:36,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:36,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,045 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:36,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:36,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:36,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:36,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2387ec5258a143b4abb6faedfe8204bc 2023-04-17 08:02:36,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:36,053 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,054 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:36,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:36,056 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:36,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:36,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:36,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:36,056 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:36,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,070 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:36,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,070 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:36,071 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:36,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:36,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,171 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:36,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:36,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:36,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:36,173 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:36,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:36,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:36,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:36,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:36,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:36,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:36,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:36,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:36,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:36,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:36,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:36,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:36,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a373304[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:36,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:36,188 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,195 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:36,195 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:36,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:36,195 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:36,203 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:36,203 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,203 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:36,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:36,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:36,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:36,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:36,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:36,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:36,206 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:36,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,212 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,213 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:36,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:36,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,220 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:36,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,220 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:36,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,221 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,321 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:36,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:36,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:36,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:36,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:36,323 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:36,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:36,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:36,323 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,324 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,328 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:36,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:36,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:36,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:36,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:36,330 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:36,330 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:36,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:36,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690d9bc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:36,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:36,338 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:36,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:36,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:36,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:36,361 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:36,361 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,361 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,361 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:36,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:36,363 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:36,363 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:36,363 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:36,363 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:36,363 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:36,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:36,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,390 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:36,390 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,391 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:36,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:36,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:36,404 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,404 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,404 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:36,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/ec759cfa9709464b8fd62353c1eeb921 2023-04-17 08:02:36,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1195567c76ae447183387e45f4884ec9 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9 2023-04-17 08:02:36,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9, entries=350, sequenceid=48, filesize=52.3 K 2023-04-17 08:02:36,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2387ec5258a143b4abb6faedfe8204bc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc 2023-04-17 08:02:36,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:02:36,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/ec759cfa9709464b8fd62353c1eeb921 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921 2023-04-17 08:02:36,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 08:02:36,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1668ms, sequenceid=48, compaction requested=true 2023-04-17 08:02:36,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:36,486 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:36,486 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:36,487 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:36,487 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:36,487 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/A is initiating minor compaction (all files) 2023-04-17 08:02:36,487 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/B is initiating minor compaction (all files) 2023-04-17 08:02:36,487 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/A in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,487 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/B in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,487 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=102.5 K 2023-04-17 08:02:36,487 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=34.8 K 2023-04-17 08:02:36,488 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1720d68a0c974517823ce24255d4a3f9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718552652 2023-04-17 08:02:36,488 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bd3cb2a7002741739dd9a423fb346376, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718552652 2023-04-17 08:02:36,489 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e710c5b696d249acaac6c6f4a9b64efb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718552665 2023-04-17 08:02:36,489 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e693b4c1373345dabdb7bb17d33d6ea2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718552665 2023-04-17 08:02:36,489 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2387ec5258a143b4abb6faedfe8204bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554811 2023-04-17 08:02:36,489 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1195567c76ae447183387e45f4884ec9, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554808 2023-04-17 08:02:36,496 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#B#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:36,502 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#A#compaction#477 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:36,504 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:36,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:36,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:36,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:36,507 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:36,507 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,507 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:36,507 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:36,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/c2de915f6da641209e34953bc488cf4e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/c2de915f6da641209e34953bc488cf4e 2023-04-17 08:02:36,508 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,508 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,511 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/B of 4972af4c15665fa4c6eba818ffa6cdb1 into c2de915f6da641209e34953bc488cf4e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:36,511 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:36,511 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/B, priority=13, startTime=1681718556486; duration=0sec 2023-04-17 08:02:36,511 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:36,511 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 08:02:36,511 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 08:02:36,512 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/C is initiating minor compaction (all files) 2023-04-17 08:02:36,512 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/C in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,512 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=34.8 K 2023-04-17 08:02:36,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:36,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:36,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:36,512 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0b98c1a06b224ca8a92fe58d52629329, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681718552652 2023-04-17 08:02:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,512 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ba4d5a8cfa404eb5b8bf1f3814b2f661, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681718552665 2023-04-17 08:02:36,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:36,513 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ec759cfa9709464b8fd62353c1eeb921, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554811 2023-04-17 08:02:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:36,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:36,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:36,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:36,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:36,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:36,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:36,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:36,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:36,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:36,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:36,513 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-17 08:02:36,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:36,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:36,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:36,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:36,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:36,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:36,519 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#C#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:36,928 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/ef0ce229519b4b7893d2d90d5f16c6ef as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ef0ce229519b4b7893d2d90d5f16c6ef 2023-04-17 08:02:36,931 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/C of 4972af4c15665fa4c6eba818ffa6cdb1 into ef0ce229519b4b7893d2d90d5f16c6ef(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:36,931 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:36,932 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/C, priority=13, startTime=1681718556486; duration=0sec 2023-04-17 08:02:36,932 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:36,948 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a419ee53a4d0423fa5afc5b33a2390b0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a419ee53a4d0423fa5afc5b33a2390b0 2023-04-17 08:02:36,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:36,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:36,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:36,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718616983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:36,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718616984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:36,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718616984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:36,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718616986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:36,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:36,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718616993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718617086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718617089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718617092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718617289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718617291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718617295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,319 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/60f201be8ee941ec8ae347c36b456dbc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/60f201be8ee941ec8ae347c36b456dbc 2023-04-17 08:02:37,322 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/A of 4972af4c15665fa4c6eba818ffa6cdb1 into 60f201be8ee941ec8ae347c36b456dbc(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:37,322 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:37,322 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/A, priority=13, startTime=1681718556486; duration=0sec 2023-04-17 08:02:37,323 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:37,352 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/a5ff8d33f84f4102815e49d501ca2004 2023-04-17 08:02:37,359 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2b5a997769184b579de28ef7775628ec 2023-04-17 08:02:37,371 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/1378e91d76bb44839d5dc0ac59faa214 2023-04-17 08:02:37,374 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/a5ff8d33f84f4102815e49d501ca2004 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004 2023-04-17 08:02:37,376 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004, entries=150, sequenceid=85, filesize=25.1 K 2023-04-17 08:02:37,377 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2b5a997769184b579de28ef7775628ec as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec 2023-04-17 08:02:37,379 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec, entries=150, sequenceid=85, filesize=11.6 K 2023-04-17 08:02:37,380 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/1378e91d76bb44839d5dc0ac59faa214 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214 2023-04-17 08:02:37,382 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214, entries=150, sequenceid=85, filesize=11.6 K 2023-04-17 08:02:37,383 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 4972af4c15665fa4c6eba818ffa6cdb1 in 870ms, sequenceid=85, compaction requested=false 2023-04-17 08:02:37,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:37,383 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:37,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:37,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:37,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:37,383 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:37,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:37,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:37,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:37,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444518cf[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:37,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:37,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:37,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:37,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:37,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:37,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:37,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:37,430 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:37,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:37,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:37,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:37,430 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:37,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:37,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,445 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:37,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:37,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:37,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:37,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:37,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:37,446 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:37,446 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:37,546 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:37,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:37,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:37,548 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:37,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:37,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:37,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:37,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:37,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:37,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:37,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:37,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:37,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:37,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:37,555 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:37,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:37,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:37,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:37,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718557609, max timestamp=1681718557612Num uniques 859; 2023-04-17 08:02:37,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718557609, max timestamp=1681718557612Num uniques 859; 2023-04-17 08:02:37,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:37,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:37,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718557609, max timestamp=1681718557612Num uniques 859; 2023-04-17 08:02:37,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718617627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718617628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718617630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718617731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718617731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718617733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718617935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718617936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718617936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:37,968 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041772f969eb5f6646a8a4e9c55269bb6b6c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041772f969eb5f6646a8a4e9c55269bb6b6c 2023-04-17 08:02:37,972 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/b6274b5e5cd14bd3a23057fca46e7724 2023-04-17 08:02:37,983 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/ab64d056b5af4b0a9a61f9304d07b0a0 2023-04-17 08:02:37,990 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/abff94bc32ae48cca3deaa6e32cc3460 2023-04-17 08:02:37,993 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/b6274b5e5cd14bd3a23057fca46e7724 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724 2023-04-17 08:02:37,995 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724, entries=150, sequenceid=97, filesize=25.1 K 2023-04-17 08:02:37,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/ab64d056b5af4b0a9a61f9304d07b0a0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0 2023-04-17 08:02:37,999 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0, entries=150, sequenceid=97, filesize=11.6 K 2023-04-17 08:02:37,999 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/abff94bc32ae48cca3deaa6e32cc3460 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460 2023-04-17 08:02:38,002 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460, entries=150, sequenceid=97, filesize=11.6 K 2023-04-17 08:02:38,002 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4972af4c15665fa4c6eba818ffa6cdb1 in 447ms, sequenceid=97, compaction requested=true 2023-04-17 08:02:38,003 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:38,003 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:38,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:38,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:38,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:38,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:38,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:38,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,037 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:38,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:38,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130b279b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:38,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:38,038 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,045 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:38,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:38,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,045 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:38,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:38,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:38,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:38,053 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:38,055 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:38,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:38,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:38,055 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:38,055 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:38,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:38,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:38,062 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:38,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:38,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:38,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,070 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:38,071 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:38,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:38,071 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:38,071 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:38,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:38,171 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:38,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:38,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:38,173 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:38,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:38,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:38,173 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:38,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,174 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:38,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:38,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:38,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,179 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:38,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:38,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:38,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:38,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:38,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:38,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:38,180 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:38,180 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:38,180 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:38,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:38,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:38,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718618245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718618246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718618249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718618350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718618350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718618351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718618552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718618552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718618554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,591 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172f6f9ce0040649a9908b862c46f0f7e7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172f6f9ce0040649a9908b862c46f0f7e7 2023-04-17 08:02:38,597 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/94569ea6648d431bbe4dd4313a7ca3ff 2023-04-17 08:02:38,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718618855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718618855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718618858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:38,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718618992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:38,994 DEBUG [Thread-3551] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:39,009 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/4bcb5decbce04bc8947f5c3d1eb585dc 2023-04-17 08:02:39,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718619013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,015 DEBUG [Thread-3547] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:39,018 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/31ea63e2730b4fb2b1eeeacf488ebeb5 2023-04-17 08:02:39,021 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/94569ea6648d431bbe4dd4313a7ca3ff as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff 2023-04-17 08:02:39,024 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff, entries=150, sequenceid=134, filesize=25.3 K 2023-04-17 08:02:39,024 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/4bcb5decbce04bc8947f5c3d1eb585dc as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc 2023-04-17 08:02:39,027 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 08:02:39,028 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/31ea63e2730b4fb2b1eeeacf488ebeb5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5 2023-04-17 08:02:39,030 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 08:02:39,031 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 4972af4c15665fa4c6eba818ffa6cdb1 in 851ms, sequenceid=134, compaction requested=true 2023-04-17 08:02:39,031 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:39,031 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:39,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:39,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:39,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:39,031 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:39,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:39,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:39,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:39,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40f789ed[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:39,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:39,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:39,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:39,070 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:39,070 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,070 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:39,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,078 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:39,078 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:39,079 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:39,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:39,080 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:39,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:39,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:39,080 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:39,080 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:39,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:39,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:39,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,088 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:39,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:39,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:39,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:39,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:39,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:39,116 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:39,116 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:39,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:39,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:39,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:39,116 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:39,116 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:39,117 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,117 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,117 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:39,117 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,217 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:39,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:39,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:39,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:39,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:39,228 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:39,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:39,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:39,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:39,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,228 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:39,237 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:39,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:39,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,237 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:39,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:39,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:39,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:39,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:39,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:39,239 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:39,239 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:39,240 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:39,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:39,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:39,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718559395, max timestamp=1681718559399Num uniques 859; 2023-04-17 08:02:39,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:39,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718559395, max timestamp=1681718559399Num uniques 859; 2023-04-17 08:02:39,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:39,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718559395, max timestamp=1681718559399Num uniques 859; 2023-04-17 08:02:39,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:39,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718619417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718619420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718619421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718619522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718619523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718619524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,650 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304174b88a779386b4c699b4413eda748d64c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174b88a779386b4c699b4413eda748d64c 2023-04-17 08:02:39,653 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/0f6180bfaf06430684a740e0bc279735 2023-04-17 08:02:39,660 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/905ba088919b4d41ba86380782e3938b 2023-04-17 08:02:39,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718619725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718619727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:39,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718619727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718620030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718620030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718620031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,068 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/48631fffd04d4d0d9ffea5281b7430e0 2023-04-17 08:02:40,071 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/0f6180bfaf06430684a740e0bc279735 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735 2023-04-17 08:02:40,074 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735, entries=150, sequenceid=144, filesize=25.3 K 2023-04-17 08:02:40,074 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/905ba088919b4d41ba86380782e3938b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b 2023-04-17 08:02:40,078 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b, entries=150, sequenceid=144, filesize=11.8 K 2023-04-17 08:02:40,079 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/48631fffd04d4d0d9ffea5281b7430e0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0 2023-04-17 08:02:40,082 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0, entries=150, sequenceid=144, filesize=11.8 K 2023-04-17 08:02:40,083 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4972af4c15665fa4c6eba818ffa6cdb1 in 844ms, sequenceid=144, compaction requested=true 2023-04-17 08:02:40,083 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:40,083 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,083 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,084 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:40,120 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,120 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,121 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:40,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402af576[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:40,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:40,122 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:40,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:40,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:40,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:40,137 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:40,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:40,139 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:40,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:40,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:40,139 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:40,139 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:40,145 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,145 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,145 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,146 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,147 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:40,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:40,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,154 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:40,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,154 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,154 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:40,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,254 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:40,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:40,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:40,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:40,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:40,257 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:40,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:40,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:40,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,257 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,262 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,262 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:40,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:40,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:40,263 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,263 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:40,264 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:40,264 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:40,293 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172dc589c3f41d4882b756f939ef199df5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172dc589c3f41d4882b756f939ef199df5 2023-04-17 08:02:40,308 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/5be324b88aa24501b844362a88661167 2023-04-17 08:02:40,321 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/f6d293a406634ff29123882d0341742c 2023-04-17 08:02:40,330 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/11989f71e3654a04b323c5fc9e3c1248 2023-04-17 08:02:40,334 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/5be324b88aa24501b844362a88661167 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167 2023-04-17 08:02:40,338 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167, entries=150, sequenceid=180, filesize=25.3 K 2023-04-17 08:02:40,339 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/f6d293a406634ff29123882d0341742c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c 2023-04-17 08:02:40,341 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c, entries=150, sequenceid=180, filesize=11.8 K 2023-04-17 08:02:40,342 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/11989f71e3654a04b323c5fc9e3c1248 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248 2023-04-17 08:02:40,344 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248, entries=150, sequenceid=180, filesize=11.8 K 2023-04-17 08:02:40,345 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 4972af4c15665fa4c6eba818ffa6cdb1 in 82ms, sequenceid=180, compaction requested=true 2023-04-17 08:02:40,345 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:40,345 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,345 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:40,353 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,353 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,354 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,354 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:40,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c451edc[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:40,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:40,355 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,362 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,362 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:40,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,362 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:40,362 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:40,362 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:40,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,370 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:40,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:40,379 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:40,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:40,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:40,379 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:40,379 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:40,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,388 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,395 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:40,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:40,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:40,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,396 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:40,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,396 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:40,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,396 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,496 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:40,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:40,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:40,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:40,504 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:40,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:40,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:40,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,504 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,512 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:40,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:40,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:40,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:40,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:40,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:40,513 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:40,513 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:40,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:40,514 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,514 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:40,520 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:40,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b8c330[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:40,522 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:40,522 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,528 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:40,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:40,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,529 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:40,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:40,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:40,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:40,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:40,539 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:40,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:40,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:40,539 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:40,539 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:40,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,575 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,575 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,575 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,575 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:40,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:40,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,576 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:40,576 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,576 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:40,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:40,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:40,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:40,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:40,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:40,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:40,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:40,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:40,595 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e27cf315dcd048f59426cc341600cb42 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e27cf315dcd048f59426cc341600cb42 2023-04-17 08:02:40,600 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=191, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/ae15e59da7b1445ca230fdebbc7818f5 2023-04-17 08:02:40,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718560599, max timestamp=1681718560623Num uniques 859; 2023-04-17 08:02:40,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718560599, max timestamp=1681718560623Num uniques 859; 2023-04-17 08:02:40,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:40,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:40,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718560599, max timestamp=1681718560623Num uniques 859; 2023-04-17 08:02:40,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718560641, max timestamp=1681718560653Num uniques 1018; 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718560641, max timestamp=1681718560653Num uniques 1018; 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718560641, max timestamp=1681718560653Num uniques 1018; 2023-04-17 08:02:40,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 08:02:40,676 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:40,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:40,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:40,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:40,680 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:40,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:40,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:40,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,680 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718620681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718620682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:40,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:40,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718620686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:40,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:40,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:40,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,692 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:40,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:40,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:40,696 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,696 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:40,704 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:40,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a7c96eb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:40,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:40,706 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,712 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:40,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:40,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:40,712 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,712 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:40,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:40,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:40,721 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:40,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:40,723 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:40,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:40,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:40,723 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:40,723 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:40,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,729 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,730 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,737 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:40,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:40,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,737 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:40,737 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,737 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718620786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718620786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718620790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,837 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:40,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:40,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:40,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:40,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:40,840 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:40,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:40,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:40,840 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,841 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:40,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:40,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:40,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:40,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:40,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:40,846 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:40,846 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,846 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:40,847 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:40,847 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,847 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:40,853 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,853 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:40,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dea5197[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:40,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:40,855 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,862 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:40,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:40,862 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:40,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:40,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:40,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:40,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:40,872 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:40,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:40,872 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:40,873 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:40,873 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:40,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:40,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:40,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,880 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:40,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:40,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:40,887 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,887 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:40,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:40,987 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:40,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:40,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:40,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:40,990 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:40,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:40,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:40,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,990 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718620989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718620989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:40,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718620994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,995 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:40,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:40,995 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,995 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:40,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:40,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:40,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:40,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:40,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:40,997 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:40,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:40,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:40,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:40,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:40,997 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:41,003 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:41,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9dd1d08[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:41,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:41,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:41,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:41,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:41,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:41,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e6fc47cf933e4c1d8df9267af2d391eb 2023-04-17 08:02:41,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:41,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:41,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:41,023 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:41,023 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:41,023 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:41,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,030 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:41,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,037 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,037 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,137 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:41,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:41,140 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:41,140 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,140 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:41,140 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:41,141 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,141 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,159 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,159 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,160 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:41,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:41,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:41,161 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,161 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:41,161 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:41,161 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:41,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:41,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29dcee40[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:41,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:41,172 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,178 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:41,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:41,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:41,179 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:41,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:41,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,187 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:41,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:41,189 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:41,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:41,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:41,189 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:41,189 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:41,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,195 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,196 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:41,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:41,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,203 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:41,204 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:41,204 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:41,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718621292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718621292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718621296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,304 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:41,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:41,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:41,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:41,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:41,307 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:41,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:41,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:41,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,307 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,312 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,312 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:41,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:41,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:41,313 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,313 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:41,313 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:41,313 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:41,313 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:41,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:41,314 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:41,320 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:41,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e9ee5e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:41,322 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:41,322 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:41,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:41,329 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:41,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,329 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:41,337 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,337 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:41,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:41,340 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:41,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:41,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:41,340 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:41,340 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:41,353 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,353 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,357 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:41,358 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,359 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:41,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:41,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,370 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:41,371 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:41,371 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:41,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:41,371 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,370 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:41,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:41,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:41,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/6c1c7a1aed364023bf35974d8a6fcd06 2023-04-17 08:02:41,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/ae15e59da7b1445ca230fdebbc7818f5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5 2023-04-17 08:02:41,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5, entries=300, sequenceid=191, filesize=45.8 K 2023-04-17 08:02:41,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e6fc47cf933e4c1d8df9267af2d391eb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb 2023-04-17 08:02:41,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb, entries=150, sequenceid=191, filesize=11.8 K 2023-04-17 08:02:41,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/6c1c7a1aed364023bf35974d8a6fcd06 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06 2023-04-17 08:02:41,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06, entries=150, sequenceid=191, filesize=11.8 K 2023-04-17 08:02:41,442 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4972af4c15665fa4c6eba818ffa6cdb1 in 864ms, sequenceid=191, compaction requested=true 2023-04-17 08:02:41,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:41,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:41,442 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 08:02:41,442 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 08:02:41,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:41,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:41,444 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 201899 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 08:02:41,444 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/A is initiating minor compaction (all files) 2023-04-17 08:02:41,444 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/A in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,444 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/60f201be8ee941ec8ae347c36b456dbc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=197.2 K 2023-04-17 08:02:41,444 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83905 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 08:02:41,444 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/B is initiating minor compaction (all files) 2023-04-17 08:02:41,444 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/B in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,444 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/c2de915f6da641209e34953bc488cf4e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=81.9 K 2023-04-17 08:02:41,444 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 60f201be8ee941ec8ae347c36b456dbc, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554811 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting c2de915f6da641209e34953bc488cf4e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554811 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting a5ff8d33f84f4102815e49d501ca2004, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718554854 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2b5a997769184b579de28ef7775628ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718554854 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting b6274b5e5cd14bd3a23057fca46e7724, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681718556981 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ab64d056b5af4b0a9a61f9304d07b0a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681718556981 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 94569ea6648d431bbe4dd4313a7ca3ff, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718557626 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4bcb5decbce04bc8947f5c3d1eb585dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718557626 2023-04-17 08:02:41,445 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 0f6180bfaf06430684a740e0bc279735, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681718558242 2023-04-17 08:02:41,446 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 905ba088919b4d41ba86380782e3938b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681718558242 2023-04-17 08:02:41,446 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5be324b88aa24501b844362a88661167, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681718559416 2023-04-17 08:02:41,446 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f6d293a406634ff29123882d0341742c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681718559416 2023-04-17 08:02:41,446 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting ae15e59da7b1445ca230fdebbc7818f5, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560558 2023-04-17 08:02:41,446 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e6fc47cf933e4c1d8df9267af2d391eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560572 2023-04-17 08:02:41,459 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#B#compaction#498 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:41,471 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:41,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:41,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:41,475 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#A#compaction#497 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:41,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:41,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:41,481 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:41,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:41,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:41,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,482 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,487 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:41,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:41,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:41,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:41,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:41,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:41,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:41,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:41,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:41,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:41,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:41,520 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:41,521 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=134.18 KB heapSize=374.77 KB 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:41,521 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:41,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:41,522 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:41,525 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/d4938a67cba44f6b8d4da74a71a3afa8 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d4938a67cba44f6b8d4da74a71a3afa8 2023-04-17 08:02:41,543 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/B of 4972af4c15665fa4c6eba818ffa6cdb1 into d4938a67cba44f6b8d4da74a71a3afa8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:41,543 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:41,543 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/B, priority=9, startTime=1681718561442; duration=0sec 2023-04-17 08:02:41,544 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:41,544 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 08:02:41,546 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83905 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 08:02:41,546 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/C is initiating minor compaction (all files) 2023-04-17 08:02:41,546 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/C in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:41,546 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ef0ce229519b4b7893d2d90d5f16c6ef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=81.9 K 2023-04-17 08:02:41,546 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting ef0ce229519b4b7893d2d90d5f16c6ef, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681718554811 2023-04-17 08:02:41,546 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1378e91d76bb44839d5dc0ac59faa214, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681718554854 2023-04-17 08:02:41,547 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting abff94bc32ae48cca3deaa6e32cc3460, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681718556981 2023-04-17 08:02:41,547 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 31ea63e2730b4fb2b1eeeacf488ebeb5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681718557626 2023-04-17 08:02:41,547 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 48631fffd04d4d0d9ffea5281b7430e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681718558242 2023-04-17 08:02:41,547 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 11989f71e3654a04b323c5fc9e3c1248, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681718559416 2023-04-17 08:02:41,548 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 6c1c7a1aed364023bf35974d8a6fcd06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560572 2023-04-17 08:02:41,588 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#C#compaction#500 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:41,602 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/d53e236b243b49b68371b97550bc93aa as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/d53e236b243b49b68371b97550bc93aa 2023-04-17 08:02:41,607 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/C of 4972af4c15665fa4c6eba818ffa6cdb1 into d53e236b243b49b68371b97550bc93aa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:41,607 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:41,607 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/C, priority=9, startTime=1681718561442; duration=0sec 2023-04-17 08:02:41,607 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:41,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:41,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:41,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718621803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718621806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718621807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718621908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718621909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718621909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:41,980 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ab75e47134c24d8f8600ee388eb1301a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ab75e47134c24d8f8600ee388eb1301a 2023-04-17 08:02:42,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718622110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718622113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718622113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,385 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/134c843e49f64b4eb82cdd88e9ec75a1 2023-04-17 08:02:42,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718622414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39318 deadline: 1681718622416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718622416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,745 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/fd0aeaba8aa945e4aa8e2456d8f48642 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/fd0aeaba8aa945e4aa8e2456d8f48642 2023-04-17 08:02:42,747 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/A of 4972af4c15665fa4c6eba818ffa6cdb1 into fd0aeaba8aa945e4aa8e2456d8f48642(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:02:42,747 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:42,748 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/A, priority=9, startTime=1681718561442; duration=1sec 2023-04-17 08:02:42,748 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:42,792 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/fc562258cbbc4a60b45606a2028cc2c6 2023-04-17 08:02:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39324 deadline: 1681718622916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39306 deadline: 1681718622917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:42,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:42,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718622918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39286 deadline: 1681718623023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,026 DEBUG [Thread-3551] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:43,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:39292 deadline: 1681718623037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,039 DEBUG [Thread-3547] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., hostname=jenkins-hbase16.apache.org,39843,1681718384756, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 08:02:43,204 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/9cea5d700cfe46969827e11c45b010e4 2023-04-17 08:02:43,209 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/134c843e49f64b4eb82cdd88e9ec75a1 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1 2023-04-17 08:02:43,211 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1, entries=150, sequenceid=243, filesize=25.3 K 2023-04-17 08:02:43,212 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/fc562258cbbc4a60b45606a2028cc2c6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6 2023-04-17 08:02:43,215 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6, entries=150, sequenceid=243, filesize=11.8 K 2023-04-17 08:02:43,216 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/9cea5d700cfe46969827e11c45b010e4 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4 2023-04-17 08:02:43,218 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4, entries=150, sequenceid=243, filesize=11.8 K 2023-04-17 08:02:43,220 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.72 KB/383712, currentSize=46.96 KB/48090 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1699ms, sequenceid=243, compaction requested=false 2023-04-17 08:02:43,220 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:43,220 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:43,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:43,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:43,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:43,220 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:43,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:43,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:43,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d07bdd[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:43,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:43,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,253 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:43,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:43,253 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,254 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:43,253 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:43,261 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:43,261 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:43,262 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:43,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:43,263 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:43,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:43,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:43,263 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:43,263 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:43,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,271 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:43,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:43,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:43,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:43,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:43,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:43,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:43,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:43,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:43,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:43,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:43,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:43,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:43,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,279 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:43,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:43,279 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:43,279 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,379 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:43,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:43,381 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:43,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:43,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:43,381 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:43,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,382 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:43,386 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,386 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:43,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:43,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:43,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:43,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:43,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:43,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:43,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:43,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:43,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:43,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:43,388 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:43,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:43,397 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173b4620dbda0945bfa846f3bf5b672a54 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173b4620dbda0945bfa846f3bf5b672a54 2023-04-17 08:02:43,803 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/e91e3dd28ef8405fb2853b48aeea5441 2023-04-17 08:02:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:43,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:43,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718563935, max timestamp=1681718563940Num uniques 859; 2023-04-17 08:02:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718563935, max timestamp=1681718563940Num uniques 859; 2023-04-17 08:02:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718563935, max timestamp=1681718563940Num uniques 859; 2023-04-17 08:02:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:43,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:43,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718623954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718623955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:43,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:43,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718623956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718624057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718624057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718624059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,217 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/71574a06bbd04d89a8f495ac157b178e 2023-04-17 08:02:44,224 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/1cb645397d26415c9c0fb14ff8b221dd 2023-04-17 08:02:44,227 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/e91e3dd28ef8405fb2853b48aeea5441 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441 2023-04-17 08:02:44,229 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441, entries=150, sequenceid=256, filesize=25.3 K 2023-04-17 08:02:44,230 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/71574a06bbd04d89a8f495ac157b178e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e 2023-04-17 08:02:44,232 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e, entries=150, sequenceid=256, filesize=11.8 K 2023-04-17 08:02:44,232 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/1cb645397d26415c9c0fb14ff8b221dd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd 2023-04-17 08:02:44,235 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd, entries=150, sequenceid=256, filesize=11.8 K 2023-04-17 08:02:44,236 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4972af4c15665fa4c6eba818ffa6cdb1 in 848ms, sequenceid=256, compaction requested=true 2023-04-17 08:02:44,236 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:44,236 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:44,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:44,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:44,236 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:44,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,245 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,245 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:44,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69840292[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:44,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:44,246 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:44,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:44,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:44,269 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,270 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:44,270 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:44,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:44,270 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:44,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718624269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718624271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718624273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,278 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:44,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:44,280 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:44,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:44,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:44,280 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:44,280 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:44,286 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,286 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,287 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,287 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,288 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,295 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:44,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:44,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,295 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,295 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,295 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718624374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718624376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718624376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,395 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:44,398 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:44,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:44,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:44,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,398 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,403 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,403 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,403 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:44,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:44,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:44,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:44,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:44,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:44,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:44,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:44,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:44,405 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:44,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:44,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:44,405 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:44,411 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,411 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:44,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27e45f0c[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:44,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:44,413 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:44,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:44,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:44,420 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:44,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:44,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,428 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:44,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:44,430 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:44,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:44,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:44,430 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:44,430 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:44,436 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,437 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,438 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,445 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,445 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,445 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:44,445 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:44,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,546 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:44,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:44,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:44,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:44,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:44,548 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:44,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:44,548 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:44,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,549 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,554 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:44,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:44,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:44,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:44,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:44,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:44,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:44,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:44,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:44,555 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:44,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:44,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:44,555 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:44,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe7deb0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:44,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:44,563 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,570 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,570 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:44,570 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:44,570 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,570 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:44,570 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:44,578 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:44,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,578 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,578 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718624577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718624579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:44,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:44,580 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:44,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:44,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:44,580 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:44,580 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:44,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718624579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,586 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,587 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,588 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:44,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,595 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,595 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,595 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:44,595 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:44,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,675 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041763d81aa8b34a473e932391d0f3b32a53 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041763d81aa8b34a473e932391d0f3b32a53 2023-04-17 08:02:44,696 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:44,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:44,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:44,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:44,698 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:44,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:44,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:44,698 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,699 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,703 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,704 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:44,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:44,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:44,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:44,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,704 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:44,705 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,705 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:44,705 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:44,705 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:44,705 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:44,711 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,711 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:44,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e25402[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:44,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:44,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:44,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:44,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:44,720 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:44,746 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:44,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,748 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:44,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:44,750 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:44,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:44,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:44,751 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:44,751 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:44,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,762 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,762 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,770 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:44,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:44,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,780 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,779 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,780 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:44,780 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:44,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,880 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:44,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718624880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:44,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:44,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:44,883 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:44,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:44,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:44,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,883 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718624882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:44,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718624883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,887 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,887 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:44,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:44,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:44,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:44,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:44,888 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:44,888 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:44,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,895 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:44,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbb409e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:44,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:44,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:44,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:44,903 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:44,903 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,903 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:44,911 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:44,911 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,911 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,912 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:44,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:44,913 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:44,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:44,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:44,914 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:44,914 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:44,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:44,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,920 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,920 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,921 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,928 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:44,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:44,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:44,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:44,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:44,928 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:44,929 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:44,929 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:44,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,029 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,032 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,032 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,033 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:45,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,038 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,038 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,038 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,038 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,038 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,045 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,045 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a26c1a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,046 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,053 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,053 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,053 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,053 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,061 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,062 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,062 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,063 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,063 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,064 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,070 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,070 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,072 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:45,078 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,078 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,078 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,079 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,079 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/afe39828e1a44c74b09ac9b909d44813 2023-04-17 08:02:45,179 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,181 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,181 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,182 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,187 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,187 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:45,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,188 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,188 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,189 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,195 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,195 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,197 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79312fe0[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,200 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,203 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,204 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,204 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,212 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,212 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,215 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,215 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,215 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,221 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,222 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,270 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,270 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,270 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,271 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,270 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,371 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,373 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,373 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,374 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718625383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718625384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,387 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,387 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:45,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,388 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,388 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,389 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718625388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,395 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,395 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd53680[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,396 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,403 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,403 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,403 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,404 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,411 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,412 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,412 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,413 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,414 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,414 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,420 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,420 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,421 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,428 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:45,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,429 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,429 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,429 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/15bd3ac628834b6e9003fe095c709a43 2023-04-17 08:02:45,529 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,532 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,532 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,537 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:45,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,538 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,539 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,539 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78cd3254[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,546 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,553 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,554 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,561 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,562 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,563 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,564 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,564 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,570 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,570 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,570 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,570 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,571 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,578 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,578 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:45,579 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,579 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,579 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,679 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,682 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,682 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,687 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,687 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:45,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,688 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,688 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,689 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,695 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,695 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136453eb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,696 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,703 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,703 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,703 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,703 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,703 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,703 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,711 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,712 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,713 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,713 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,713 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,720 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,720 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,721 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:45,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,747 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,747 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,747 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,747 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,748 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,748 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,748 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,748 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,848 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:45,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:45,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:45,850 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:45,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,850 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:45,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:45,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,851 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,861 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,862 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:45,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:45,863 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,863 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:45,863 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:45,863 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:45,863 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:45,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:45,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635a2048[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:45,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:45,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:45,878 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,878 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:45,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:45,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:45,886 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:45,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,886 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:45,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:45,888 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:45,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:45,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:45,889 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:45,889 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:45,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/a603021018484d209cf7c4e605339b4f 2023-04-17 08:02:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/afe39828e1a44c74b09ac9b909d44813 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813 2023-04-17 08:02:45,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813, entries=200, sequenceid=293, filesize=32.3 K 2023-04-17 08:02:45,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/15bd3ac628834b6e9003fe095c709a43 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:45,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:45,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:45,903 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:45,903 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:45,904 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:45,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:45,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:45,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:45,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:45,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:45,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43, entries=150, sequenceid=293, filesize=11.9 K 2023-04-17 08:02:45,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/a603021018484d209cf7c4e605339b4f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f 2023-04-17 08:02:45,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f, entries=150, sequenceid=293, filesize=11.9 K 2023-04-17 08:02:45,910 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1645ms, sequenceid=293, compaction requested=true 2023-04-17 08:02:45,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:45,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:45,910 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:45,910 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:45,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:45,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:45,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48636 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:45,911 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111116 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:45,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/B is initiating minor compaction (all files) 2023-04-17 08:02:45,911 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/A is initiating minor compaction (all files) 2023-04-17 08:02:45,911 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/B in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,911 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/A in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:45,911 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d4938a67cba44f6b8d4da74a71a3afa8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=47.5 K 2023-04-17 08:02:45,911 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/fd0aeaba8aa945e4aa8e2456d8f48642, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=108.5 K 2023-04-17 08:02:45,911 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d4938a67cba44f6b8d4da74a71a3afa8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560572 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting fd0aeaba8aa945e4aa8e2456d8f48642, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560572 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting fc562258cbbc4a60b45606a2028cc2c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681718560666 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 134c843e49f64b4eb82cdd88e9ec75a1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681718560666 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 71574a06bbd04d89a8f495ac157b178e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718561803 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting e91e3dd28ef8405fb2853b48aeea5441, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718561803 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 15bd3ac628834b6e9003fe095c709a43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563954 2023-04-17 08:02:45,912 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting afe39828e1a44c74b09ac9b909d44813, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563951 2023-04-17 08:02:45,918 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#B#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:45,920 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#A#compaction#509 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:46,004 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:46,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:46,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:46,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:46,006 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:46,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:46,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:46,006 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:46,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:46,011 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,011 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:46,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:46,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:46,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:46,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:46,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:46,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:46,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:46,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:46,013 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:46,013 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:46,013 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:46,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:46,014 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:46,184 DEBUG [regionserver/jenkins-hbase16:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.07692307692307693, tune throughput to 53.85 MB/second 2023-04-17 08:02:46,325 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/b0a7dee8b6e14567ba5ab384919d2d57 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b0a7dee8b6e14567ba5ab384919d2d57 2023-04-17 08:02:46,329 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/B of 4972af4c15665fa4c6eba818ffa6cdb1 into b0a7dee8b6e14567ba5ab384919d2d57(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:46,329 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:46,329 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/B, priority=12, startTime=1681718565910; duration=0sec 2023-04-17 08:02:46,329 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:46,330 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:46,331 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48636 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:46,331 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/C is initiating minor compaction (all files) 2023-04-17 08:02:46,331 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/C in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:46,331 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/d53e236b243b49b68371b97550bc93aa, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=47.5 K 2023-04-17 08:02:46,331 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting d53e236b243b49b68371b97550bc93aa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681718560572 2023-04-17 08:02:46,332 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 9cea5d700cfe46969827e11c45b010e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1681718560666 2023-04-17 08:02:46,332 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1cb645397d26415c9c0fb14ff8b221dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681718561803 2023-04-17 08:02:46,332 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting a603021018484d209cf7c4e605339b4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563954 2023-04-17 08:02:46,339 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#C#compaction#512 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:46,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:46,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718566412, max timestamp=1681718566419Num uniques 859; 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718566412, max timestamp=1681718566419Num uniques 859; 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718566412, max timestamp=1681718566419Num uniques 859; 2023-04-17 08:02:46,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:46,423 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173aecd7fe327e466a8c642428911abbdf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173aecd7fe327e466a8c642428911abbdf 2023-04-17 08:02:46,426 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/2673e3e8b18144378653b2f41ff889e6 2023-04-17 08:02:46,440 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/4ccd49ce8be0481bb7dbc290f8113791 2023-04-17 08:02:46,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718566437, max timestamp=1681718566447Num uniques 1068; 2023-04-17 08:02:46,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-17 08:02:46,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718566437, max timestamp=1681718566447Num uniques 1068; 2023-04-17 08:02:46,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681718566437, max timestamp=1681718566447Num uniques 1068; 2023-04-17 08:02:46,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-17 08:02:46,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-17 08:02:46,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718626482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718626483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718626484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718626585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718626585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718626587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,734 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/4a6dadab4eaa4b3d9a52102cec861bbe as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4a6dadab4eaa4b3d9a52102cec861bbe 2023-04-17 08:02:46,737 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/A of 4972af4c15665fa4c6eba818ffa6cdb1 into 4a6dadab4eaa4b3d9a52102cec861bbe(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:46,737 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:46,737 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/A, priority=12, startTime=1681718565910; duration=0sec 2023-04-17 08:02:46,737 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:46,745 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/3793f40969114bebaff88acbc5be06ce as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/3793f40969114bebaff88acbc5be06ce 2023-04-17 08:02:46,748 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/C of 4972af4c15665fa4c6eba818ffa6cdb1 into 3793f40969114bebaff88acbc5be06ce(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:46,748 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:46,748 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/C, priority=12, startTime=1681718565910; duration=0sec 2023-04-17 08:02:46,748 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:46,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718626788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718626788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:46,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718626790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,853 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/797bebc2df734f3c8257a957f9bf52fd 2023-04-17 08:02:46,856 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/2673e3e8b18144378653b2f41ff889e6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6 2023-04-17 08:02:46,859 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6, entries=100, sequenceid=302, filesize=18.6 K 2023-04-17 08:02:46,859 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/4ccd49ce8be0481bb7dbc290f8113791 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791 2023-04-17 08:02:46,862 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791, entries=100, sequenceid=302, filesize=9.5 K 2023-04-17 08:02:46,862 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/797bebc2df734f3c8257a957f9bf52fd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd 2023-04-17 08:02:46,864 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd, entries=100, sequenceid=302, filesize=9.5 K 2023-04-17 08:02:46,865 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=142.33 KB/145742 for 4972af4c15665fa4c6eba818ffa6cdb1 in 852ms, sequenceid=302, compaction requested=false 2023-04-17 08:02:46,865 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:46,865 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:46,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:46,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:46,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:46,865 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:46,870 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:46,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:46,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,870 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:46,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:46,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15cd6b95[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:46,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:46,871 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,878 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:46,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:46,878 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:46,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,879 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:46,886 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,886 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:46,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:46,887 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:46,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:46,888 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:46,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:46,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:46,888 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:46,888 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:46,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:46,895 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,896 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:46,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:46,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:46,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:46,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:46,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:46,903 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:46,903 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:46,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:46,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:46,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:46,904 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:46,903 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:46,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:46,904 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:47,004 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:47,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:47,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:47,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:47,006 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:47,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:47,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:47,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:47,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,007 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,056 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:47,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:47,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,056 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:47,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:47,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:47,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:47,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:47,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:47,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:47,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:47,057 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:47,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:47,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:47,057 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=142.33 KB heapSize=402.63 KB 2023-04-17 08:02:47,057 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:47,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:47,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:47,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:47,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:47,058 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:47,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:47,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:47,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718627095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718627096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718627097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718627198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718627199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718627200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718627401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718627402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718627402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,467 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173293fa8b3a8249ddb27d142b6ae83f70 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173293fa8b3a8249ddb27d142b6ae83f70 2023-04-17 08:02:47,470 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/adabf54499984ef3b8f5086c7fc47cf7 2023-04-17 08:02:47,478 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/88b8f7bd43914ad0b70e35a414de7309 2023-04-17 08:02:47,486 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.93 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e0eda428913947aaa5b545cf3dde8ee5 2023-04-17 08:02:47,489 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/adabf54499984ef3b8f5086c7fc47cf7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7 2023-04-17 08:02:47,492 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7, entries=150, sequenceid=359, filesize=25.4 K 2023-04-17 08:02:47,492 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/88b8f7bd43914ad0b70e35a414de7309 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309 2023-04-17 08:02:47,496 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 08:02:47,496 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e0eda428913947aaa5b545cf3dde8ee5 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5 2023-04-17 08:02:47,499 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 08:02:47,499 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~142.33 KB/145742, heapSize ~402.58 KB/412240, currentSize=40.25 KB/41220 for 4972af4c15665fa4c6eba818ffa6cdb1 in 442ms, sequenceid=359, compaction requested=true 2023-04-17 08:02:47,500 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:47,500 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:47,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:47,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:47,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:47,500 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:47,528 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,528 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:47,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:47,529 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:47,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:47,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403a4ea[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:47,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:47,530 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,536 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:47,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:47,537 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:47,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,537 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:47,545 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:47,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:47,545 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:47,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:47,547 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:47,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:47,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:47,547 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:47,547 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:47,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:47,553 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:47,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,553 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:47,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,555 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,561 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,561 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:47,561 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:47,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:47,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:47,562 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:47,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:47,562 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,562 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:47,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:47,562 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:47,662 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:47,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:47,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:47,665 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:47,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:47,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:47,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:47,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,665 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,690 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:47,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:47,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,690 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:47,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:47,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:47,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:47,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:47,691 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:47,691 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:47,691 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:47,691 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:47,691 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:47,691 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:47,692 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:47,692 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:47,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:47,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:47,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718567710, max timestamp=1681718567720Num uniques 859; 2023-04-17 08:02:47,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:47,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718567710, max timestamp=1681718567720Num uniques 859; 2023-04-17 08:02:47,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718567710, max timestamp=1681718567720Num uniques 859; 2023-04-17 08:02:47,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:47,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:47,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718627743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718627743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718627744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718627847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718627847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:47,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718627847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718628049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718628050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718628050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,103 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417435c77312f9b4987a2288541768e0cf9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417435c77312f9b4987a2288541768e0cf9 2023-04-17 08:02:48,106 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=368, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1ca46de1319647308076cc98ba94ca0e 2023-04-17 08:02:48,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718628352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718628354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718628355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,513 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/7cb775a47d7c45129ed59138c619ccdb 2023-04-17 08:02:48,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718628855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718628858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:48,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718628858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,920 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e2848d258fb84a4294c129c3a4ea88c3 2023-04-17 08:02:48,923 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/1ca46de1319647308076cc98ba94ca0e as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e 2023-04-17 08:02:48,925 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e, entries=150, sequenceid=368, filesize=25.4 K 2023-04-17 08:02:48,926 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/7cb775a47d7c45129ed59138c619ccdb as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb 2023-04-17 08:02:48,929 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 08:02:48,929 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e2848d258fb84a4294c129c3a4ea88c3 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3 2023-04-17 08:02:48,931 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 08:02:48,932 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1240ms, sequenceid=368, compaction requested=true 2023-04-17 08:02:48,932 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:48,932 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:48,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:48,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:48,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:48,932 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:48,961 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,961 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:48,962 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:48,962 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:48,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:48,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3549448a[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:48,963 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:48,963 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:48,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:48,970 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:48,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,970 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:48,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:48,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:48,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:48,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:48,980 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:48,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:48,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:48,980 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:48,980 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:48,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,987 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:48,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:48,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:48,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,988 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:48,995 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:48,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:48,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:48,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:48,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:48,995 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:48,995 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:48,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:48,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:49,095 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:49,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:49,098 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:49,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:49,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:49,098 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:49,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:49,099 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:49,103 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,103 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:49,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:49,103 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,103 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:49,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:49,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:49,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:49,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:49,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:49,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:49,104 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:49,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:49,105 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:49,105 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:49,105 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:49,115 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417bf0eac2e21bb44488c30b506135af118 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bf0eac2e21bb44488c30b506135af118 2023-04-17 08:02:49,119 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/7bada57c5ec043159eb77849f61858cf 2023-04-17 08:02:49,548 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/03cf5690906b46ad8c4433022d3c1bfd 2023-04-17 08:02:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:49,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:49,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:49,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718629870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718629871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718629874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,956 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e853fdf714e64237817144de2b08ded0 2023-04-17 08:02:49,959 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/7bada57c5ec043159eb77849f61858cf as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf 2023-04-17 08:02:49,962 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf, entries=150, sequenceid=405, filesize=25.4 K 2023-04-17 08:02:49,962 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/03cf5690906b46ad8c4433022d3c1bfd as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd 2023-04-17 08:02:49,965 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 08:02:49,965 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e853fdf714e64237817144de2b08ded0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0 2023-04-17 08:02:49,967 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 08:02:49,968 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 4972af4c15665fa4c6eba818ffa6cdb1 in 863ms, sequenceid=405, compaction requested=true 2023-04-17 08:02:49,968 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:49,968 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:49,968 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:49,968 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:49,968 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,968 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:49,968 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:49,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 08:02:49,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:49,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:49,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:49,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:49,978 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:49,978 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:49,979 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:49,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:49,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7237a0b[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:49,981 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:49,981 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,985 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d2712aea38214ae583a183711b8ad105 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d2712aea38214ae583a183711b8ad105 2023-04-17 08:02:49,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:49,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:49,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:49,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:49,987 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:49,989 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/76c50b084fae4ce984692f1c49a0453c 2023-04-17 08:02:49,995 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:49,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:49,995 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:49,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2de92d75f95d48f882e8380c702b476b 2023-04-17 08:02:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,004 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,004 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,004 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718569992, max timestamp=1681718570002Num uniques 859; 2023-04-17 08:02:50,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:02:50,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681718569996, max timestamp=1681718570002Num uniques 859; 2023-04-17 08:02:50,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 08:02:50,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681718569992, max timestamp=1681718570002Num uniques 859; 2023-04-17 08:02:50,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 08:02:50,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,012 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,013 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,014 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,020 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,020 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,020 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/f8bd950250ff4a85aa8b47699e98b715 2023-04-17 08:02:50,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/76c50b084fae4ce984692f1c49a0453c as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c 2023-04-17 08:02:50,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c, entries=150, sequenceid=415, filesize=25.4 K 2023-04-17 08:02:50,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/2de92d75f95d48f882e8380c702b476b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b 2023-04-17 08:02:50,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718630029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 08:02:50,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/f8bd950250ff4a85aa8b47699e98b715 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715 2023-04-17 08:02:50,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 08:02:50,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 4972af4c15665fa4c6eba818ffa6cdb1 in 61ms, sequenceid=415, compaction requested=true 2023-04-17 08:02:50,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:50,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:50,038 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:02:50,038 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:02:50,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:50,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:50,039 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 149706 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:02:50,039 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/A is initiating minor compaction (all files) 2023-04-17 08:02:50,040 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/A in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,040 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71150 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:02:50,040 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4a6dadab4eaa4b3d9a52102cec861bbe, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=146.2 K 2023-04-17 08:02:50,040 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/B is initiating minor compaction (all files) 2023-04-17 08:02:50,040 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/B in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,040 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b0a7dee8b6e14567ba5ab384919d2d57, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=69.5 K 2023-04-17 08:02:50,040 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 4a6dadab4eaa4b3d9a52102cec861bbe, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563954 2023-04-17 08:02:50,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:50,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-04-17 08:02:50,040 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b0a7dee8b6e14567ba5ab384919d2d57, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563954 2023-04-17 08:02:50,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:50,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:50,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:50,040 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 2673e3e8b18144378653b2f41ff889e6, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681718564268 2023-04-17 08:02:50,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 4ccd49ce8be0481bb7dbc290f8113791, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681718564268 2023-04-17 08:02:50,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:50,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting adabf54499984ef3b8f5086c7fc47cf7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681718566472 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 88b8f7bd43914ad0b70e35a414de7309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681718566472 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 1ca46de1319647308076cc98ba94ca0e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718567090 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 7cb775a47d7c45129ed59138c619ccdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718567090 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 7bada57c5ec043159eb77849f61858cf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681718567738 2023-04-17 08:02:50,041 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 03cf5690906b46ad8c4433022d3c1bfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681718567738 2023-04-17 08:02:50,042 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 76c50b084fae4ce984692f1c49a0453c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:50,042 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 2de92d75f95d48f882e8380c702b476b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:50,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718630050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,057 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:50,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718630054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,094 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#A#compaction#528 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:50,098 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178519a569c5f74eab94cfd0436a0d67d3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178519a569c5f74eab94cfd0436a0d67d3 2023-04-17 08:02:50,121 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,123 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,123 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,124 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,128 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,128 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,129 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:50,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,129 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,129 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,129 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,130 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:50,130 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:50,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:50,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:50,130 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:50,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718630134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,136 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,136 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,137 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:50,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cc28b9[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:50,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:50,138 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,145 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:50,145 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:50,145 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:50,145 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:50,153 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:50,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,153 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,153 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,155 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,155 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,155 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718630155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,161 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,161 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718630159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,161 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,162 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,170 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,170 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,270 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,273 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,273 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,278 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,278 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:50,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,279 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,279 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,279 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:50,280 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:50,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:50,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:50,280 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:50,308 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,308 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,308 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,308 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:50,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6071f79e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:50,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:50,309 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,320 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:50,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:50,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,320 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:50,320 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:50,328 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,328 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,330 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,330 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,330 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718630335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,345 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,345 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,345 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,345 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718630357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718630363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,445 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,448 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,449 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,453 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,453 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:50,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,454 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,455 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,455 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,455 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:50,455 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:50,455 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:50,461 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,461 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,462 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:50,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f63920e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:50,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:50,463 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,470 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:50,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:50,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,470 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:50,470 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:50,478 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:50,478 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,478 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,478 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,480 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,480 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,480 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,486 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,486 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,486 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,487 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,488 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,496 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,496 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,497 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,497 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,497 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,499 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/b1b03583a6f040d6837c8f13b02284c6 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b1b03583a6f040d6837c8f13b02284c6 2023-04-17 08:02:50,502 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/36f48629b93a4523ae1feab41a9aaeac 2023-04-17 08:02:50,503 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/B of 4972af4c15665fa4c6eba818ffa6cdb1 into b1b03583a6f040d6837c8f13b02284c6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:50,503 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:50,503 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/B, priority=10, startTime=1681718570038; duration=0sec 2023-04-17 08:02:50,503 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:50,503 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 08:02:50,505 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71150 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 08:02:50,507 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/C is initiating minor compaction (all files) 2023-04-17 08:02:50,507 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/C in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,507 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/3793f40969114bebaff88acbc5be06ce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=69.5 K 2023-04-17 08:02:50,507 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 3793f40969114bebaff88acbc5be06ce, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681718563954 2023-04-17 08:02:50,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 797bebc2df734f3c8257a957f9bf52fd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681718564268 2023-04-17 08:02:50,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e0eda428913947aaa5b545cf3dde8ee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681718566472 2023-04-17 08:02:50,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e2848d258fb84a4294c129c3a4ea88c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681718567090 2023-04-17 08:02:50,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e853fdf714e64237817144de2b08ded0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681718567738 2023-04-17 08:02:50,508 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f8bd950250ff4a85aa8b47699e98b715, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:50,515 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#C#compaction#531 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:50,597 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,601 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,601 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,611 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,611 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,612 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,613 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,616 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,616 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,616 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:50,616 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:50,616 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:50,628 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,628 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,628 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,628 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c50cb[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:50,630 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:50,630 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,637 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:50,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:50,637 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,637 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:50,637 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:50,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718630641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,645 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:50,645 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,645 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,645 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,648 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,648 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,648 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,649 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,649 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,653 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,654 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,656 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,662 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,662 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,662 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,662 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718630662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718630667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,763 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,769 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,769 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,769 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,769 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,770 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,770 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,778 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,778 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,779 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:50,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,780 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,781 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,781 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,781 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,781 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,781 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:50,782 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:50,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:50,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:50,782 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:50,786 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,786 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:50,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1d29e3[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:50,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:50,789 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,795 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:50,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:50,795 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:50,795 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,795 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:50,825 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,825 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,825 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,826 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:50,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:50,828 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:50,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:50,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:50,828 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:50,828 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:50,836 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,837 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:50,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,837 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,839 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:50,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:50,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:50,845 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:50,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,846 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:50,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:50,846 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/bd6c5aefd50d4c04b684ab0c3135e376 2023-04-17 08:02:50,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/81ed4ceb30374e79a617eed5ca8413e0 2023-04-17 08:02:50,922 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e406220b55ec4c43ad504a1812e29738 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e406220b55ec4c43ad504a1812e29738 2023-04-17 08:02:50,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/36f48629b93a4523ae1feab41a9aaeac as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac 2023-04-17 08:02:50,927 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/C of 4972af4c15665fa4c6eba818ffa6cdb1 into e406220b55ec4c43ad504a1812e29738(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:50,927 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:50,927 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/C, priority=10, startTime=1681718570038; duration=0sec 2023-04-17 08:02:50,927 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:50,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac, entries=200, sequenceid=452, filesize=32.3 K 2023-04-17 08:02:50,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/bd6c5aefd50d4c04b684ab0c3135e376 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376 2023-04-17 08:02:50,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376, entries=150, sequenceid=452, filesize=11.9 K 2023-04-17 08:02:50,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/81ed4ceb30374e79a617eed5ca8413e0 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0 2023-04-17 08:02:50,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0, entries=150, sequenceid=452, filesize=11.9 K 2023-04-17 08:02:50,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~405.25 KB/414976, currentSize=40.25 KB/41220 for 4972af4c15665fa4c6eba818ffa6cdb1 in 901ms, sequenceid=452, compaction requested=false 2023-04-17 08:02:50,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:50,946 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:50,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:50,948 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:50,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:50,948 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:50,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,949 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,953 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,953 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:50,953 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:50,953 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:50,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:50,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:50,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:50,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:50,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:50,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:50,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:50,955 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:50,955 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:50,955 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:50,972 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a069f402379b4078927951e9af40457f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a069f402379b4078927951e9af40457f 2023-04-17 08:02:51,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:51,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:51,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718571169, max timestamp=1681718571175Num uniques 859; 2023-04-17 08:02:51,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:51,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718571169, max timestamp=1681718571175Num uniques 859; 2023-04-17 08:02:51,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681718571169, max timestamp=1681718571175Num uniques 859; 2023-04-17 08:02:51,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:51,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:51,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718571189, max timestamp=1681718571194Num uniques 1068; 2023-04-17 08:02:51,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-17 08:02:51,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718571189, max timestamp=1681718571194Num uniques 1068; 2023-04-17 08:02:51,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-17 08:02:51,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681718571189, max timestamp=1681718571194Num uniques 1068; 2023-04-17 08:02:51,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-17 08:02:51,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718631215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718631215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718631217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,312 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/f06b6cab193449b0a37cf89ecb078110 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/f06b6cab193449b0a37cf89ecb078110 2023-04-17 08:02:51,315 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/A of 4972af4c15665fa4c6eba818ffa6cdb1 into f06b6cab193449b0a37cf89ecb078110(size=26.1 K), total size for store is 58.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 08:02:51,315 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:51,315 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/A, priority=10, startTime=1681718570038; duration=1sec 2023-04-17 08:02:51,316 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:51,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718631319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718631319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718631320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,377 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=463, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/5712a8d2352e4dceba7da2bf2e506187 2023-04-17 08:02:51,388 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/1fa86e9541654310a1996adfb19e846a 2023-04-17 08:02:51,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718631521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718631521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718631522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,794 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/0d34785b42c8459fafe24eb7a85bc35d 2023-04-17 08:02:51,798 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/5712a8d2352e4dceba7da2bf2e506187 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187 2023-04-17 08:02:51,800 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187, entries=150, sequenceid=463, filesize=25.4 K 2023-04-17 08:02:51,801 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/1fa86e9541654310a1996adfb19e846a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a 2023-04-17 08:02:51,803 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a, entries=150, sequenceid=463, filesize=11.9 K 2023-04-17 08:02:51,804 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/0d34785b42c8459fafe24eb7a85bc35d as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d 2023-04-17 08:02:51,806 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d, entries=150, sequenceid=463, filesize=11.9 K 2023-04-17 08:02:51,807 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4972af4c15665fa4c6eba818ffa6cdb1 in 851ms, sequenceid=463, compaction requested=true 2023-04-17 08:02:51,807 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:51,807 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:51,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:51,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:51,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:51,807 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:51,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-17 08:02:51,828 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:51,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 08:02:51,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:51,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 08:02:51,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:51,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 08:02:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718631833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718631833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718631833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,836 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:51,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:51,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,837 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:51,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:51,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bff019[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:51,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:51,838 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,845 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,845 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:51,845 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:51,845 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:51,845 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417da3f7e9269c94c6f98f8d05e28d3f2e6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da3f7e9269c94c6f98f8d05e28d3f2e6 2023-04-17 08:02:51,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,845 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:51,853 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,853 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:51,853 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=519, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/3be6cd0d5eed47c198b1741c37a37396 2023-04-17 08:02:51,853 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:51,854 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:51,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:51,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:51,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:51,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:51,858 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:51,858 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:51,861 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,861 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:51,861 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,862 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:51,863 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:51,875 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:51,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:51,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:51,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,875 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:51,875 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:51,875 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:51,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:51,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:51,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e6c25835ac4d4cf6a82ea39fd142cac7 2023-04-17 08:02:51,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718631936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718631936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718631936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,975 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:51,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:51,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:51,977 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:51,977 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:51,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:51,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:51,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,978 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:51,986 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,986 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:51,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,987 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:51,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:51,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:51,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:51,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:51,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:51,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:51,996 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:51,996 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:52,003 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,003 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,004 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:52,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cfff62[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:52,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:52,005 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,011 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:52,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:52,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,012 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:52,012 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:52,020 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:52,022 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:52,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:52,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:52,022 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:52,022 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:52,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,028 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:52,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:52,029 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,030 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,036 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:52,036 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:52,037 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,037 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:52,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:52,037 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:52,037 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:52,137 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:52,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:52,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:52,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39306 deadline: 1681718632139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39318 deadline: 1681718632140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 08:02:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:39324 deadline: 1681718632140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:52,144 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:52,144 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:52,144 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:52,144 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:52,145 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,145 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,153 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:52,153 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:52,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,154 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:52,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:52,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:52,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:52,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:52,154 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:52,155 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,155 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:52,155 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:52,155 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:52,155 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:52,161 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,161 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,162 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:52,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1624f358[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:52,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:52,163 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,170 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:52,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:52,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,170 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:52,170 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:52,178 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:52,178 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:52,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:52,180 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:52,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:52,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:52,180 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:52,180 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:52,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,204 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:52,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:52,205 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,206 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,220 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:52,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:52,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:52,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:52,220 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:52,221 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:52,221 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:52,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:52,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/f704f68eb0ea40d783eb381a2ef2dc3f 2023-04-17 08:02:52,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/3be6cd0d5eed47c198b1741c37a37396 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396 2023-04-17 08:02:52,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396, entries=250, sequenceid=519, filesize=39.2 K 2023-04-17 08:02:52,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/e6c25835ac4d4cf6a82ea39fd142cac7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7 2023-04-17 08:02:52,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7, entries=150, sequenceid=519, filesize=11.9 K 2023-04-17 08:02:52,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/f704f68eb0ea40d783eb381a2ef2dc3f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f 2023-04-17 08:02:52,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f, entries=150, sequenceid=519, filesize=11.9 K 2023-04-17 08:02:52,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~435.19 KB/445632, currentSize=26.84 KB/27480 for 4972af4c15665fa4c6eba818ffa6cdb1 in 484ms, sequenceid=519, compaction requested=true 2023-04-17 08:02:52,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:52,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:52,313 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:52,313 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:52,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:52,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:52,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:52,314 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:52,314 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/A is initiating minor compaction (all files) 2023-04-17 08:02:52,315 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/B is initiating minor compaction (all files) 2023-04-17 08:02:52,315 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/A in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,315 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/B in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,315 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/f06b6cab193449b0a37cf89ecb078110, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=123.0 K 2023-04-17 08:02:52,315 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b1b03583a6f040d6837c8f13b02284c6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=48.3 K 2023-04-17 08:02:52,315 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting f06b6cab193449b0a37cf89ecb078110, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:52,315 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting b1b03583a6f040d6837c8f13b02284c6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 36f48629b93a4523ae1feab41a9aaeac, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681718570026 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting bd6c5aefd50d4c04b684ab0c3135e376, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681718570031 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 5712a8d2352e4dceba7da2bf2e506187, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1681718570045 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 1fa86e9541654310a1996adfb19e846a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1681718570045 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] compactions.Compactor(207): Compacting 3be6cd0d5eed47c198b1741c37a37396, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681718571207 2023-04-17 08:02:52,316 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e6c25835ac4d4cf6a82ea39fd142cac7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681718571207 2023-04-17 08:02:52,321 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:52,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:52,324 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#B#compaction#539 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:52,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-17 08:02:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 08:02:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 08:02:52,327 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 08:02:52,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:52,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 08:02:52,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 08:02:52,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,327 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,336 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#A#compaction#540 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:52,336 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:52,336 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:52,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:52,337 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 08:02:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 08:02:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:52,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 08:02:52,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 08:02:52,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 08:02:52,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 08:02:52,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 08:02:52,340 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 08:02:52,341 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/a123485b75a4453d89a1b9471b64a23f as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/a123485b75a4453d89a1b9471b64a23f 2023-04-17 08:02:52,346 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/B of 4972af4c15665fa4c6eba818ffa6cdb1 into a123485b75a4453d89a1b9471b64a23f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:52,346 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:52,346 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/B, priority=12, startTime=1681718572313; duration=0sec 2023-04-17 08:02:52,346 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 08:02:52,347 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 08:02:52,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 08:02:52,347 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 08:02:52,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. started... 2023-04-17 08:02:52,347 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 08:02:52,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:52,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:52,347 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:52,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:52,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:52,348 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 08:02:52,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 08:02:52,348 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1912): 4972af4c15665fa4c6eba818ffa6cdb1/C is initiating minor compaction (all files) 2023-04-17 08:02:52,348 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4972af4c15665fa4c6eba818ffa6cdb1/C in TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:52,348 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e406220b55ec4c43ad504a1812e29738, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f] into tmpdir=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp, totalSize=48.3 K 2023-04-17 08:02:52,349 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting e406220b55ec4c43ad504a1812e29738, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681718569869 2023-04-17 08:02:52,349 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 81ed4ceb30374e79a617eed5ca8413e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681718570031 2023-04-17 08:02:52,349 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting 0d34785b42c8459fafe24eb7a85bc35d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1681718570045 2023-04-17 08:02:52,350 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] compactions.Compactor(207): Compacting f704f68eb0ea40d783eb381a2ef2dc3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681718571207 2023-04-17 08:02:52,361 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] throttle.PressureAwareThroughputController(145): 4972af4c15665fa4c6eba818ffa6cdb1#C#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-17 08:02:52,392 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e8a91e96bb2048f49f95ecad8dfa6f24 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e8a91e96bb2048f49f95ecad8dfa6f24 2023-04-17 08:02:52,393 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304175836c5a5f4784cd9893b5721f00463ba to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175836c5a5f4784cd9893b5721f00463ba 2023-04-17 08:02:52,396 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/C of 4972af4c15665fa4c6eba818ffa6cdb1 into e8a91e96bb2048f49f95ecad8dfa6f24(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:52,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:52,396 INFO [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/C, priority=12, startTime=1681718572313; duration=0sec 2023-04-17 08:02:52,396 DEBUG [RS:0;jenkins-hbase16:39843-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:52,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] regionserver.HRegion(9158): Flush requested on 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:52,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. as already flushing 2023-04-17 08:02:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718572445, max timestamp=1681718572460Num uniques 859; 2023-04-17 08:02:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718572445, max timestamp=1681718572460Num uniques 859; 2023-04-17 08:02:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681718572445, max timestamp=1681718572460Num uniques 859; 2023-04-17 08:02:52,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 08:02:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681718572475, max timestamp=1681718572485Num uniques 1068; 2023-04-17 08:02:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 08:02:52,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681718572475, max timestamp=1681718572485Num uniques 1068; 2023-04-17 08:02:52,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 08:02:52,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681718572475, max timestamp=1681718572485Num uniques 1068; 2023-04-17 08:02:52,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 08:02:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718572490, max timestamp=1681718572528Num uniques 1077; 2023-04-17 08:02:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 08:02:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718572490, max timestamp=1681718572528Num uniques 1077; 2023-04-17 08:02:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681718572490, max timestamp=1681718572528Num uniques 1077; 2023-04-17 08:02:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 08:02:52,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 08:02:52,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718572549, max timestamp=1681718572555Num uniques 1086; 2023-04-17 08:02:52,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-17 08:02:52,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718572549, max timestamp=1681718572555Num uniques 1086; 2023-04-17 08:02:52,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-17 08:02:52,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681718572549, max timestamp=1681718572555Num uniques 1086; 2023-04-17 08:02:52,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718572588, max timestamp=1681718572597Num uniques 1045; 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718572588, max timestamp=1681718572598Num uniques 1045; 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681718572588, max timestamp=1681718572597Num uniques 1045; 2023-04-17 08:02:52,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681718572613, max timestamp=1681718572623Num uniques 1054; 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=11624, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681718572613, max timestamp=1681718572623Num uniques 1054; 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=11624, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681718572613, max timestamp=1681718572623Num uniques 1054; 2023-04-17 08:02:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=11624, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 08:02:52,627 DEBUG [Thread-3562] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3da68fe0 to 127.0.0.1:62477 2023-04-17 08:02:52,627 DEBUG [Thread-3562] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,628 DEBUG [Thread-3558] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x387217bb to 127.0.0.1:62477 2023-04-17 08:02:52,628 DEBUG [Thread-3558] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,628 DEBUG [Thread-3556] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b8bf14 to 127.0.0.1:62477 2023-04-17 08:02:52,628 DEBUG [Thread-3556] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,629 DEBUG [Thread-3549] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f11e84a to 127.0.0.1:62477 2023-04-17 08:02:52,629 DEBUG [Thread-3549] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,629 DEBUG [Thread-3545] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5c51ec to 127.0.0.1:62477 2023-04-17 08:02:52,629 DEBUG [Thread-3560] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c00a18c to 127.0.0.1:62477 2023-04-17 08:02:52,629 DEBUG [Thread-3545] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,629 DEBUG [Thread-3560] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,629 DEBUG [Thread-3553] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4143b9ae to 127.0.0.1:62477 2023-04-17 08:02:52,629 DEBUG [Thread-3553] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,630 DEBUG [Thread-3564] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fd90ac8 to 127.0.0.1:62477 2023-04-17 08:02:52,630 DEBUG [Thread-3564] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:52,802 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/4af0476e65724f608bc9f90158509040 2023-04-17 08:02:53,030 DEBUG [Thread-3551] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a1010a6 to 127.0.0.1:62477 2023-04-17 08:02:53,031 DEBUG [Thread-3551] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:53,071 DEBUG [Thread-3547] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5be55982 to 127.0.0.1:62477 2023-04-17 08:02:53,071 DEBUG [Thread-3547] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:53,164 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/d8066796562f45eb9334115f2cde1c05 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/d8066796562f45eb9334115f2cde1c05 2023-04-17 08:02:53,167 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4972af4c15665fa4c6eba818ffa6cdb1/A of 4972af4c15665fa4c6eba818ffa6cdb1 into d8066796562f45eb9334115f2cde1c05(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 08:02:53,167 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:53,167 INFO [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1., storeName=4972af4c15665fa4c6eba818ffa6cdb1/A, priority=12, startTime=1681718572313; duration=0sec 2023-04-17 08:02:53,168 DEBUG [RS:0;jenkins-hbase16:39843-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 08:02:53,211 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/5670bda7ce074f648d45f5a4c2e18e5a 2023-04-17 08:02:53,623 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e49a8299ce584b1f8fd9e897e5971a3b 2023-04-17 08:02:53,627 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/4af0476e65724f608bc9f90158509040 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4af0476e65724f608bc9f90158509040 2023-04-17 08:02:53,630 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4af0476e65724f608bc9f90158509040, entries=150, sequenceid=527, filesize=25.4 K 2023-04-17 08:02:53,631 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/5670bda7ce074f648d45f5a4c2e18e5a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/5670bda7ce074f648d45f5a4c2e18e5a 2023-04-17 08:02:53,634 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/5670bda7ce074f648d45f5a4c2e18e5a, entries=150, sequenceid=527, filesize=11.9 K 2023-04-17 08:02:53,635 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/e49a8299ce584b1f8fd9e897e5971a3b as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e49a8299ce584b1f8fd9e897e5971a3b 2023-04-17 08:02:53,638 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e49a8299ce584b1f8fd9e897e5971a3b, entries=150, sequenceid=527, filesize=11.9 K 2023-04-17 08:02:53,638 INFO [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=73.80 KB/75570 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1291ms, sequenceid=527, compaction requested=false 2023-04-17 08:02:53,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:53,639 DEBUG [rs(jenkins-hbase16.apache.org,39843,1681718384756)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:53,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 08:02:53,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 08:02:53,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 08:02:53,639 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 08:02:53,680 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,680 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:53,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,681 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 08:02:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:53,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39843,1681718384756' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 08:02:53,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6404824e[Count = 0] remaining members to acquire global barrier 2023-04-17 08:02:53,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 08:02:53,684 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,714 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 08:02:53,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 08:02:53,715 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,39843,1681718384756' in zk 2023-04-17 08:02:53,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,715 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 08:02:53,728 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 08:02:53,728 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,728 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:53,728 DEBUG [member: 'jenkins-hbase16.apache.org,39843,1681718384756' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 08:02:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,39843,1681718384756': 2023-04-17 08:02:53,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39843,1681718384756' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 08:02:53,731 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 08:02:53,731 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 08:02:53,731 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 08:02:53,731 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 08:02:53,731 INFO [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 08:02:53,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 08:02:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 08:02:53,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,763 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 08:02:53,763 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 08:02:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,765 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 08:02:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 08:02:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,802 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 08:02:53,802 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,802 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:53,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 08:02:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 08:02:53,802 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 08:02:53,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 08:02:53,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 08:02:53,803 DEBUG [(jenkins-hbase16.apache.org,34569,1681718383610)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 08:02:53,803 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 08:02:53,803 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,803 DEBUG [Thread-3555] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 08:02:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 08:02:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 08:02:53,803 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 08:02:53,804 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 08:02:53,803 DEBUG [Thread-3555] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 08:02:53,904 DEBUG [Thread-3555] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 08:02:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 179 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 174 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-17 08:02:53,905 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(394): wrote 171 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5270 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5491 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5116 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5294 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(398): read 5469 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 08:02:53,906 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:02:53,906 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c217430 to 127.0.0.1:62477 2023-04-17 08:02:53,906 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:53,907 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 08:02:53,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-17 08:02:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:53,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:53,910 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718573910"}]},"ts":"1681718573910"} 2023-04-17 08:02:53,910 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 08:02:53,928 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 08:02:53,929 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, UNASSIGN}] 2023-04-17 08:02:53,937 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, UNASSIGN 2023-04-17 08:02:53,938 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:53,938 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718573938"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681718573938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681718573938"}]},"ts":"1681718573938"} 2023-04-17 08:02:53,939 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756}] 2023-04-17 08:02:54,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:54,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:54,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 4972af4c15665fa4c6eba818ffa6cdb1, disabling compactions & flushes 2023-04-17 08:02:54,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. after waiting 0 ms 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:54,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 4972af4c15665fa4c6eba818ffa6cdb1 3/3 column families, dataSize=73.80 KB heapSize=196.78 KB 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=A 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=B 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4972af4c15665fa4c6eba818ffa6cdb1, store=C 2023-04-17 08:02:54,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 08:02:54,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:54,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ee7957da77384e10a63c8fc9194f6290 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ee7957da77384e10a63c8fc9194f6290 2023-04-17 08:02:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:54,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=641, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/b00e04f0c377498a850cc5cdac69f952 2023-04-17 08:02:54,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/d224babf0c354c31b345d391d9369786 2023-04-17 08:02:55,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:55,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/4c5e0f1c9abc4c61903e180f910be282 2023-04-17 08:02:55,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/A/b00e04f0c377498a850cc5cdac69f952 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b00e04f0c377498a850cc5cdac69f952 2023-04-17 08:02:55,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b00e04f0c377498a850cc5cdac69f952, entries=150, sequenceid=641, filesize=25.4 K 2023-04-17 08:02:55,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/B/d224babf0c354c31b345d391d9369786 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d224babf0c354c31b345d391d9369786 2023-04-17 08:02:55,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d224babf0c354c31b345d391d9369786, entries=150, sequenceid=641, filesize=11.9 K 2023-04-17 08:02:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/.tmp/C/4c5e0f1c9abc4c61903e180f910be282 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/4c5e0f1c9abc4c61903e180f910be282 2023-04-17 08:02:55,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/4c5e0f1c9abc4c61903e180f910be282, entries=150, sequenceid=641, filesize=11.9 K 2023-04-17 08:02:55,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~196.73 KB/201456, currentSize=0 B/0 for 4972af4c15665fa4c6eba818ffa6cdb1 in 1261ms, sequenceid=641, compaction requested=true 2023-04-17 08:02:55,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/60f201be8ee941ec8ae347c36b456dbc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/fd0aeaba8aa945e4aa8e2456d8f48642, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4a6dadab4eaa4b3d9a52102cec861bbe, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/f06b6cab193449b0a37cf89ecb078110, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396] to archive 2023-04-17 08:02:55,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:55,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1720d68a0c974517823ce24255d4a3f9 2023-04-17 08:02:55,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e693b4c1373345dabdb7bb17d33d6ea2 2023-04-17 08:02:55,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1195567c76ae447183387e45f4884ec9 2023-04-17 08:02:55,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/60f201be8ee941ec8ae347c36b456dbc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/60f201be8ee941ec8ae347c36b456dbc 2023-04-17 08:02:55,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/a5ff8d33f84f4102815e49d501ca2004 2023-04-17 08:02:55,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b6274b5e5cd14bd3a23057fca46e7724 2023-04-17 08:02:55,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/94569ea6648d431bbe4dd4313a7ca3ff 2023-04-17 08:02:55,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/0f6180bfaf06430684a740e0bc279735 2023-04-17 08:02:55,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5be324b88aa24501b844362a88661167 2023-04-17 08:02:55,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/ae15e59da7b1445ca230fdebbc7818f5 2023-04-17 08:02:55,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/fd0aeaba8aa945e4aa8e2456d8f48642 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/fd0aeaba8aa945e4aa8e2456d8f48642 2023-04-17 08:02:55,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/134c843e49f64b4eb82cdd88e9ec75a1 2023-04-17 08:02:55,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/e91e3dd28ef8405fb2853b48aeea5441 2023-04-17 08:02:55,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/afe39828e1a44c74b09ac9b909d44813 2023-04-17 08:02:55,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4a6dadab4eaa4b3d9a52102cec861bbe to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4a6dadab4eaa4b3d9a52102cec861bbe 2023-04-17 08:02:55,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/2673e3e8b18144378653b2f41ff889e6 2023-04-17 08:02:55,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/adabf54499984ef3b8f5086c7fc47cf7 2023-04-17 08:02:55,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/1ca46de1319647308076cc98ba94ca0e 2023-04-17 08:02:55,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/7bada57c5ec043159eb77849f61858cf 2023-04-17 08:02:55,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/f06b6cab193449b0a37cf89ecb078110 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/f06b6cab193449b0a37cf89ecb078110 2023-04-17 08:02:55,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/76c50b084fae4ce984692f1c49a0453c 2023-04-17 08:02:55,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/36f48629b93a4523ae1feab41a9aaeac 2023-04-17 08:02:55,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/5712a8d2352e4dceba7da2bf2e506187 2023-04-17 08:02:55,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/3be6cd0d5eed47c198b1741c37a37396 2023-04-17 08:02:55,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/c2de915f6da641209e34953bc488cf4e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d4938a67cba44f6b8d4da74a71a3afa8, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b0a7dee8b6e14567ba5ab384919d2d57, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b1b03583a6f040d6837c8f13b02284c6, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7] to archive 2023-04-17 08:02:55,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:55,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd3cb2a7002741739dd9a423fb346376 2023-04-17 08:02:55,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e710c5b696d249acaac6c6f4a9b64efb 2023-04-17 08:02:55,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/c2de915f6da641209e34953bc488cf4e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/c2de915f6da641209e34953bc488cf4e 2023-04-17 08:02:55,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2387ec5258a143b4abb6faedfe8204bc 2023-04-17 08:02:55,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2b5a997769184b579de28ef7775628ec 2023-04-17 08:02:55,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/ab64d056b5af4b0a9a61f9304d07b0a0 2023-04-17 08:02:55,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4bcb5decbce04bc8947f5c3d1eb585dc 2023-04-17 08:02:55,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/905ba088919b4d41ba86380782e3938b 2023-04-17 08:02:55,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/f6d293a406634ff29123882d0341742c 2023-04-17 08:02:55,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d4938a67cba44f6b8d4da74a71a3afa8 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d4938a67cba44f6b8d4da74a71a3afa8 2023-04-17 08:02:55,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6fc47cf933e4c1d8df9267af2d391eb 2023-04-17 08:02:55,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/fc562258cbbc4a60b45606a2028cc2c6 2023-04-17 08:02:55,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/71574a06bbd04d89a8f495ac157b178e 2023-04-17 08:02:55,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b0a7dee8b6e14567ba5ab384919d2d57 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b0a7dee8b6e14567ba5ab384919d2d57 2023-04-17 08:02:55,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/15bd3ac628834b6e9003fe095c709a43 2023-04-17 08:02:55,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/4ccd49ce8be0481bb7dbc290f8113791 2023-04-17 08:02:55,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/88b8f7bd43914ad0b70e35a414de7309 2023-04-17 08:02:55,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/7cb775a47d7c45129ed59138c619ccdb 2023-04-17 08:02:55,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/03cf5690906b46ad8c4433022d3c1bfd 2023-04-17 08:02:55,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b1b03583a6f040d6837c8f13b02284c6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/b1b03583a6f040d6837c8f13b02284c6 2023-04-17 08:02:55,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/2de92d75f95d48f882e8380c702b476b 2023-04-17 08:02:55,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/bd6c5aefd50d4c04b684ab0c3135e376 2023-04-17 08:02:55,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/1fa86e9541654310a1996adfb19e846a 2023-04-17 08:02:55,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/e6c25835ac4d4cf6a82ea39fd142cac7 2023-04-17 08:02:55,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ef0ce229519b4b7893d2d90d5f16c6ef, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/d53e236b243b49b68371b97550bc93aa, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/3793f40969114bebaff88acbc5be06ce, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e406220b55ec4c43ad504a1812e29738, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f] to archive 2023-04-17 08:02:55,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 08:02:55,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0b98c1a06b224ca8a92fe58d52629329 2023-04-17 08:02:55,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ba4d5a8cfa404eb5b8bf1f3814b2f661 2023-04-17 08:02:55,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ef0ce229519b4b7893d2d90d5f16c6ef to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ef0ce229519b4b7893d2d90d5f16c6ef 2023-04-17 08:02:55,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/ec759cfa9709464b8fd62353c1eeb921 2023-04-17 08:02:55,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1378e91d76bb44839d5dc0ac59faa214 2023-04-17 08:02:55,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/abff94bc32ae48cca3deaa6e32cc3460 2023-04-17 08:02:55,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/31ea63e2730b4fb2b1eeeacf488ebeb5 2023-04-17 08:02:55,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/48631fffd04d4d0d9ffea5281b7430e0 2023-04-17 08:02:55,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/11989f71e3654a04b323c5fc9e3c1248 2023-04-17 08:02:55,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/d53e236b243b49b68371b97550bc93aa to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/d53e236b243b49b68371b97550bc93aa 2023-04-17 08:02:55,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/6c1c7a1aed364023bf35974d8a6fcd06 2023-04-17 08:02:55,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/9cea5d700cfe46969827e11c45b010e4 2023-04-17 08:02:55,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/1cb645397d26415c9c0fb14ff8b221dd 2023-04-17 08:02:55,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/3793f40969114bebaff88acbc5be06ce to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/3793f40969114bebaff88acbc5be06ce 2023-04-17 08:02:55,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/a603021018484d209cf7c4e605339b4f 2023-04-17 08:02:55,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/797bebc2df734f3c8257a957f9bf52fd 2023-04-17 08:02:55,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e0eda428913947aaa5b545cf3dde8ee5 2023-04-17 08:02:55,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e2848d258fb84a4294c129c3a4ea88c3 2023-04-17 08:02:55,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e853fdf714e64237817144de2b08ded0 2023-04-17 08:02:55,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e406220b55ec4c43ad504a1812e29738 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e406220b55ec4c43ad504a1812e29738 2023-04-17 08:02:55,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f8bd950250ff4a85aa8b47699e98b715 2023-04-17 08:02:55,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/81ed4ceb30374e79a617eed5ca8413e0 2023-04-17 08:02:55,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/0d34785b42c8459fafe24eb7a85bc35d 2023-04-17 08:02:55,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/f704f68eb0ea40d783eb381a2ef2dc3f 2023-04-17 08:02:55,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits/644.seqid, newMaxSeqId=644, maxSeqId=4 2023-04-17 08:02:55,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1. 2023-04-17 08:02:55,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 4972af4c15665fa4c6eba818ffa6cdb1: 2023-04-17 08:02:55,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:55,442 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4972af4c15665fa4c6eba818ffa6cdb1, regionState=CLOSED 2023-04-17 08:02:55,442 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681718575442"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681718575442"}]},"ts":"1681718575442"} 2023-04-17 08:02:55,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-17 08:02:55,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 4972af4c15665fa4c6eba818ffa6cdb1, server=jenkins-hbase16.apache.org,39843,1681718384756 in 1.5040 sec 2023-04-17 08:02:55,444 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-17 08:02:55,444 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4972af4c15665fa4c6eba818ffa6cdb1, UNASSIGN in 1.5140 sec 2023-04-17 08:02:55,445 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681718575445"}]},"ts":"1681718575445"} 2023-04-17 08:02:55,445 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 08:02:55,478 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 08:02:55,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5720 sec 2023-04-17 08:02:56,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 08:02:56,014 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-17 08:02:56,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-17 08:02:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,017 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,017 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 08:02:56,023 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:56,024 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C, FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits] 2023-04-17 08:02:56,026 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4af0476e65724f608bc9f90158509040 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/4af0476e65724f608bc9f90158509040 2023-04-17 08:02:56,027 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b00e04f0c377498a850cc5cdac69f952 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/b00e04f0c377498a850cc5cdac69f952 2023-04-17 08:02:56,028 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/d8066796562f45eb9334115f2cde1c05 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/A/d8066796562f45eb9334115f2cde1c05 2023-04-17 08:02:56,030 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/5670bda7ce074f648d45f5a4c2e18e5a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/5670bda7ce074f648d45f5a4c2e18e5a 2023-04-17 08:02:56,031 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/a123485b75a4453d89a1b9471b64a23f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/a123485b75a4453d89a1b9471b64a23f 2023-04-17 08:02:56,032 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d224babf0c354c31b345d391d9369786 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/B/d224babf0c354c31b345d391d9369786 2023-04-17 08:02:56,034 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/4c5e0f1c9abc4c61903e180f910be282 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/4c5e0f1c9abc4c61903e180f910be282 2023-04-17 08:02:56,034 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e49a8299ce584b1f8fd9e897e5971a3b to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e49a8299ce584b1f8fd9e897e5971a3b 2023-04-17 08:02:56,035 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e8a91e96bb2048f49f95ecad8dfa6f24 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/C/e8a91e96bb2048f49f95ecad8dfa6f24 2023-04-17 08:02:56,037 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits/644.seqid to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1/recovered.edits/644.seqid 2023-04-17 08:02:56,037 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/.tmp/data/default/TestAcidGuarantees/4972af4c15665fa4c6eba818ffa6cdb1 2023-04-17 08:02:56,038 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 08:02:56,038 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:02:56,038 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 08:02:56,040 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172dc589c3f41d4882b756f939ef199df5 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172dc589c3f41d4882b756f939ef199df5 2023-04-17 08:02:56,041 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172f6f9ce0040649a9908b862c46f0f7e7 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172f6f9ce0040649a9908b862c46f0f7e7 2023-04-17 08:02:56,042 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173293fa8b3a8249ddb27d142b6ae83f70 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173293fa8b3a8249ddb27d142b6ae83f70 2023-04-17 08:02:56,043 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173aecd7fe327e466a8c642428911abbdf to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173aecd7fe327e466a8c642428911abbdf 2023-04-17 08:02:56,044 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173b4620dbda0945bfa846f3bf5b672a54 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173b4620dbda0945bfa846f3bf5b672a54 2023-04-17 08:02:56,045 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417435c77312f9b4987a2288541768e0cf9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417435c77312f9b4987a2288541768e0cf9 2023-04-17 08:02:56,047 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174b88a779386b4c699b4413eda748d64c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174b88a779386b4c699b4413eda748d64c 2023-04-17 08:02:56,048 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e9906b541a74fdba20074b57e4797ff to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174e9906b541a74fdba20074b57e4797ff 2023-04-17 08:02:56,049 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175836c5a5f4784cd9893b5721f00463ba to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175836c5a5f4784cd9893b5721f00463ba 2023-04-17 08:02:56,050 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175c61546012c04fe4ae97d74d0b9ed6f9 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175c61546012c04fe4ae97d74d0b9ed6f9 2023-04-17 08:02:56,051 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041763d81aa8b34a473e932391d0f3b32a53 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041763d81aa8b34a473e932391d0f3b32a53 2023-04-17 08:02:56,053 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176faf167d5cb6424b86b56bfbb704958c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176faf167d5cb6424b86b56bfbb704958c 2023-04-17 08:02:56,054 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041772f969eb5f6646a8a4e9c55269bb6b6c to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041772f969eb5f6646a8a4e9c55269bb6b6c 2023-04-17 08:02:56,055 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178519a569c5f74eab94cfd0436a0d67d3 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178519a569c5f74eab94cfd0436a0d67d3 2023-04-17 08:02:56,057 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a069f402379b4078927951e9af40457f to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a069f402379b4078927951e9af40457f 2023-04-17 08:02:56,058 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a419ee53a4d0423fa5afc5b33a2390b0 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a419ee53a4d0423fa5afc5b33a2390b0 2023-04-17 08:02:56,060 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ab75e47134c24d8f8600ee388eb1301a to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ab75e47134c24d8f8600ee388eb1301a 2023-04-17 08:02:56,061 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bf0eac2e21bb44488c30b506135af118 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bf0eac2e21bb44488c30b506135af118 2023-04-17 08:02:56,063 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d2712aea38214ae583a183711b8ad105 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d2712aea38214ae583a183711b8ad105 2023-04-17 08:02:56,064 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da3f7e9269c94c6f98f8d05e28d3f2e6 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da3f7e9269c94c6f98f8d05e28d3f2e6 2023-04-17 08:02:56,065 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e27cf315dcd048f59426cc341600cb42 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e27cf315dcd048f59426cc341600cb42 2023-04-17 08:02:56,067 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ee7957da77384e10a63c8fc9194f6290 to hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ee7957da77384e10a63c8fc9194f6290 2023-04-17 08:02:56,067 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 08:02:56,069 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,070 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 08:02:56,071 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 08:02:56,072 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,072 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 08:02:56,072 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681718576072"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:56,073 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 08:02:56,073 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4972af4c15665fa4c6eba818ffa6cdb1, NAME => 'TestAcidGuarantees,,1681718549355.4972af4c15665fa4c6eba818ffa6cdb1.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 08:02:56,073 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 08:02:56,073 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681718576073"}]},"ts":"9223372036854775807"} 2023-04-17 08:02:56,074 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 08:02:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 08:02:56,123 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 08:02:56,124 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 108 msec 2023-04-17 08:02:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34569] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 08:02:56,320 INFO [Listener at localhost.localdomain/40407] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-17 08:02:56,329 INFO [Listener at localhost.localdomain/40407] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=545 (was 535) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=310 (was 274) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=8440 (was 7694) - AvailableMemoryMB LEAK? - 2023-04-17 08:02:56,330 INFO [Listener at localhost.localdomain/40407] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-17 08:02:56,330 INFO [Listener at localhost.localdomain/40407] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 08:02:56,330 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ddc537c to 127.0.0.1:62477 2023-04-17 08:02:56,330 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:56,330 DEBUG [Listener at localhost.localdomain/40407] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-17 08:02:56,330 DEBUG [Listener at localhost.localdomain/40407] util.JVMClusterUtil(257): Found active master hash=632975141, stopped=false 2023-04-17 08:02:56,330 INFO [Listener at localhost.localdomain/40407] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 08:02:56,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 08:02:56,369 INFO [Listener at localhost.localdomain/40407] procedure2.ProcedureExecutor(629): Stopping 2023-04-17 08:02:56,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 08:02:56,369 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 08:02:56,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 08:02:56,369 DEBUG [Listener at localhost.localdomain/40407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1839b432 to 127.0.0.1:62477 2023-04-17 08:02:56,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 08:02:56,369 DEBUG [Listener at localhost.localdomain/40407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:56,370 INFO [Listener at localhost.localdomain/40407] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,39843,1681718384756' ***** 2023-04-17 08:02:56,370 INFO [Listener at localhost.localdomain/40407] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-17 08:02:56,370 INFO [RS:0;jenkins-hbase16:39843] regionserver.HeapMemoryManager(220): Stopping 2023-04-17 08:02:56,370 INFO [RS:0;jenkins-hbase16:39843] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-17 08:02:56,370 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-17 08:02:56,370 INFO [RS:0;jenkins-hbase16:39843] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-17 08:02:56,370 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(3304): Received CLOSE for c90e072c3dc123157787d896ec6fad3d 2023-04-17 08:02:56,371 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:56,372 DEBUG [RS:0;jenkins-hbase16:39843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x712c8aa1 to 127.0.0.1:62477 2023-04-17 08:02:56,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing c90e072c3dc123157787d896ec6fad3d, disabling compactions & flushes 2023-04-17 08:02:56,372 DEBUG [RS:0;jenkins-hbase16:39843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:56,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 08:02:56,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 08:02:56,372 INFO [RS:0;jenkins-hbase16:39843] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-17 08:02:56,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. after waiting 0 ms 2023-04-17 08:02:56,372 INFO [RS:0;jenkins-hbase16:39843] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-17 08:02:56,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 08:02:56,372 INFO [RS:0;jenkins-hbase16:39843] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-17 08:02:56,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing c90e072c3dc123157787d896ec6fad3d 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-17 08:02:56,372 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-17 08:02:56,372 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-17 08:02:56,373 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 08:02:56,373 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1479): Online Regions={c90e072c3dc123157787d896ec6fad3d=hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d., 1588230740=hbase:meta,,1.1588230740} 2023-04-17 08:02:56,373 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 08:02:56,373 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 08:02:56,373 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 08:02:56,373 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 08:02:56,373 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-17 08:02:56,374 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1505): Waiting on 1588230740, c90e072c3dc123157787d896ec6fad3d 2023-04-17 08:02:56,378 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-17 08:02:56,379 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/info/0648c6ebec734037ba525f92a08c2cc7 2023-04-17 08:02:56,384 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0648c6ebec734037ba525f92a08c2cc7 2023-04-17 08:02:56,575 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1505): Waiting on 1588230740, c90e072c3dc123157787d896ec6fad3d 2023-04-17 08:02:56,775 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1505): Waiting on 1588230740, c90e072c3dc123157787d896ec6fad3d 2023-04-17 08:02:56,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/.tmp/info/691c11a1d16b4f929ce7360b86bb6c12 2023-04-17 08:02:56,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/.tmp/info/691c11a1d16b4f929ce7360b86bb6c12 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/info/691c11a1d16b4f929ce7360b86bb6c12 2023-04-17 08:02:56,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/info/691c11a1d16b4f929ce7360b86bb6c12, entries=2, sequenceid=6, filesize=4.8 K 2023-04-17 08:02:56,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c90e072c3dc123157787d896ec6fad3d in 415ms, sequenceid=6, compaction requested=false 2023-04-17 08:02:56,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/namespace/c90e072c3dc123157787d896ec6fad3d/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-17 08:02:56,791 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/rep_barrier/b0b3d8949d104df1b83e0c0e93a3dc6a 2023-04-17 08:02:56,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 08:02:56,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for c90e072c3dc123157787d896ec6fad3d: 2023-04-17 08:02:56,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681718387120.c90e072c3dc123157787d896ec6fad3d. 2023-04-17 08:02:56,794 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b0b3d8949d104df1b83e0c0e93a3dc6a 2023-04-17 08:02:56,975 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 08:02:57,156 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-17 08:02:57,157 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-17 08:02:57,176 DEBUG [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 08:02:57,201 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/table/bd48a5491d1e410d9b5a0c6fc7f82f99 2023-04-17 08:02:57,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd48a5491d1e410d9b5a0c6fc7f82f99 2023-04-17 08:02:57,210 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/info/0648c6ebec734037ba525f92a08c2cc7 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/info/0648c6ebec734037ba525f92a08c2cc7 2023-04-17 08:02:57,217 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0648c6ebec734037ba525f92a08c2cc7 2023-04-17 08:02:57,217 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/info/0648c6ebec734037ba525f92a08c2cc7, entries=22, sequenceid=93, filesize=7.1 K 2023-04-17 08:02:57,220 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/rep_barrier/b0b3d8949d104df1b83e0c0e93a3dc6a as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/rep_barrier/b0b3d8949d104df1b83e0c0e93a3dc6a 2023-04-17 08:02:57,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b0b3d8949d104df1b83e0c0e93a3dc6a 2023-04-17 08:02:57,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/rep_barrier/b0b3d8949d104df1b83e0c0e93a3dc6a, entries=6, sequenceid=93, filesize=5.5 K 2023-04-17 08:02:57,227 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/.tmp/table/bd48a5491d1e410d9b5a0c6fc7f82f99 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/table/bd48a5491d1e410d9b5a0c6fc7f82f99 2023-04-17 08:02:57,232 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd48a5491d1e410d9b5a0c6fc7f82f99 2023-04-17 08:02:57,232 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/table/bd48a5491d1e410d9b5a0c6fc7f82f99, entries=9, sequenceid=93, filesize=5.5 K 2023-04-17 08:02:57,233 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 859ms, sequenceid=93, compaction requested=false 2023-04-17 08:02:57,238 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-17 08:02:57,238 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 08:02:57,240 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 08:02:57,240 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 08:02:57,240 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-17 08:02:57,376 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,39843,1681718384756; all regions closed. 2023-04-17 08:02:57,383 DEBUG [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/oldWALs 2023-04-17 08:02:57,383 INFO [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C39843%2C1681718384756.meta:.meta(num 1681718386844) 2023-04-17 08:02:57,387 DEBUG [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/oldWALs 2023-04-17 08:02:57,388 INFO [RS:0;jenkins-hbase16:39843] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C39843%2C1681718384756:(num 1681718386284) 2023-04-17 08:02:57,388 DEBUG [RS:0;jenkins-hbase16:39843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:57,388 INFO [RS:0;jenkins-hbase16:39843] regionserver.LeaseManager(133): Closed leases 2023-04-17 08:02:57,388 INFO [RS:0;jenkins-hbase16:39843] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-17 08:02:57,388 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 08:02:57,389 INFO [RS:0;jenkins-hbase16:39843] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:39843 2023-04-17 08:02:57,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 08:02:57,403 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,39843,1681718384756 2023-04-17 08:02:57,403 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3147eceb rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2216] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,404 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 08:02:57,404 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f6e1ee9 rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2216] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,411 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,39843,1681718384756] 2023-04-17 08:02:57,412 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,39843,1681718384756; numProcessing=1 2023-04-17 08:02:57,420 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,39843,1681718384756 already deleted, retry=false 2023-04-17 08:02:57,420 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,39843,1681718384756 expired; onlineServers=0 2023-04-17 08:02:57,420 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,34569,1681718383610' ***** 2023-04-17 08:02:57,420 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-17 08:02:57,420 DEBUG [M:0;jenkins-hbase16:34569] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6487651e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-04-17 08:02:57,420 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,34569,1681718383610 2023-04-17 08:02:57,420 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,34569,1681718383610; all regions closed. 2023-04-17 08:02:57,420 DEBUG [M:0;jenkins-hbase16:34569] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 08:02:57,420 DEBUG [M:0;jenkins-hbase16:34569] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-17 08:02:57,420 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-17 08:02:57,420 DEBUG [M:0;jenkins-hbase16:34569] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-17 08:02:57,420 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1681718385958] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1681718385958,5,FailOnTimeoutGroup] 2023-04-17 08:02:57,420 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1681718385959] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1681718385959,5,FailOnTimeoutGroup] 2023-04-17 08:02:57,421 INFO [M:0;jenkins-hbase16:34569] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-17 08:02:57,421 INFO [M:0;jenkins-hbase16:34569] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-17 08:02:57,421 INFO [M:0;jenkins-hbase16:34569] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-04-17 08:02:57,421 DEBUG [M:0;jenkins-hbase16:34569] master.HMaster(1512): Stopping service threads 2023-04-17 08:02:57,421 INFO [M:0;jenkins-hbase16:34569] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-17 08:02:57,422 ERROR [M:0;jenkins-hbase16:34569] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (794240053) connection to localhost.localdomain/127.0.0.1:46073 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-17 08:02:57,422 INFO [M:0;jenkins-hbase16:34569] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-17 08:02:57,422 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-17 08:02:57,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-17 08:02:57,428 DEBUG [M:0;jenkins-hbase16:34569] zookeeper.ZKUtil(398): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-17 08:02:57,428 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 08:02:57,428 WARN [M:0;jenkins-hbase16:34569] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-17 08:02:57,428 INFO [M:0;jenkins-hbase16:34569] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-17 08:02:57,428 INFO [M:0;jenkins-hbase16:34569] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-17 08:02:57,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 08:02:57,430 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 08:02:57,430 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 08:02:57,430 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 08:02:57,430 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 08:02:57,430 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 08:02:57,430 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.29 KB heapSize=393.94 KB 2023-04-17 08:02:57,512 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 08:02:57,512 INFO [RS:0;jenkins-hbase16:39843] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,39843,1681718384756; zookeeper connection closed. 2023-04-17 08:02:57,513 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@37be35de rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2216] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,513 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3b5b9e3a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3b5b9e3a 2023-04-17 08:02:57,513 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): regionserver:39843-0x10196577f520001, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 08:02:57,513 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f73c42a rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2216] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,514 INFO [Listener at localhost.localdomain/40407] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-17 08:02:57,841 INFO [M:0;jenkins-hbase16:34569] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.29 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4e30a1728e374c9c8823dddf1739b652 2023-04-17 08:02:57,845 INFO [M:0;jenkins-hbase16:34569] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4e30a1728e374c9c8823dddf1739b652 2023-04-17 08:02:57,846 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4e30a1728e374c9c8823dddf1739b652 as hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4e30a1728e374c9c8823dddf1739b652 2023-04-17 08:02:57,848 INFO [M:0;jenkins-hbase16:34569] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4e30a1728e374c9c8823dddf1739b652 2023-04-17 08:02:57,849 INFO [M:0;jenkins-hbase16:34569] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46073/user/jenkins/test-data/c9e22bb2-9d5a-435a-cdf2-bac093df6b36/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4e30a1728e374c9c8823dddf1739b652, entries=65, sequenceid=578, filesize=24.7 K 2023-04-17 08:02:57,849 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegion(2948): Finished flush of dataSize ~345.29 KB/353574, heapSize ~393.92 KB/403376, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-17 08:02:57,850 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 08:02:57,850 DEBUG [M:0;jenkins-hbase16:34569] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 08:02:57,853 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 08:02:57,853 INFO [M:0;jenkins-hbase16:34569] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-17 08:02:57,853 INFO [M:0;jenkins-hbase16:34569] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:34569 2023-04-17 08:02:57,861 DEBUG [M:0;jenkins-hbase16:34569] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,34569,1681718383610 already deleted, retry=false 2023-04-17 08:02:57,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 08:02:57,970 INFO [M:0;jenkins-hbase16:34569] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,34569,1681718383610; zookeeper connection closed. 2023-04-17 08:02:57,970 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@32fc7e74 rejected from java.util.concurrent.ThreadPoolExecutor@4d7ac027[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3709] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,970 DEBUG [Listener at localhost.localdomain/40407-EventThread] zookeeper.ZKWatcher(600): master:34569-0x10196577f520000, quorum=127.0.0.1:62477, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 08:02:57,970 ERROR [Listener at localhost.localdomain/40407-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1cc056f2 rejected from java.util.concurrent.ThreadPoolExecutor@4d7ac027[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3709] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 08:02:57,972 WARN [Listener at localhost.localdomain/40407] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-17 08:02:57,999 INFO [Listener at localhost.localdomain/40407] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-17 08:02:58,106 WARN [BP-1048812278-188.40.62.62-1681718380168 heartbeating to localhost.localdomain/127.0.0.1:46073] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-17 08:02:58,106 WARN [BP-1048812278-188.40.62.62-1681718380168 heartbeating to localhost.localdomain/127.0.0.1:46073] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1048812278-188.40.62.62-1681718380168 (Datanode Uuid 372b2a2d-65cf-4d91-a317-aff28cf3703e) service to localhost.localdomain/127.0.0.1:46073 2023-04-17 08:02:58,109 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18/dfs/data/data1/current/BP-1048812278-188.40.62.62-1681718380168] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 08:02:58,110 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0be0dd1-8e74-e240-c248-d83a0e76835d/cluster_41d28004-9d04-8675-b4e1-bf9ff7a27b18/dfs/data/data2/current/BP-1048812278-188.40.62.62-1681718380168] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 08:02:58,141 INFO [Listener at localhost.localdomain/40407] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-17 08:02:58,310 INFO [Listener at localhost.localdomain/40407] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-17 08:02:58,476 INFO [Listener at localhost.localdomain/40407] hbase.HBaseTestingUtility(1293): Minicluster is down